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ABSTRACT  d  ■  ^ 


There  is  an  increasing  requirement  from  high  levels 
within  the  Government  that  the  Navy's  aircraft  cost 
estimators  and  analysts  provide  explicit  estimates  for  the 
sub-elements  of  Aircraft  System  Test  and  Evaluation  (AST&E) 
efforts.  The  data  required  to  produce  more  accurate  and 
detailed  estimates  represent  lower  levels  in  the  Aircraft 
Work  Breakdown  Structure  (WBS)  than  previously  available. 
This  is  a  two  volume  thesis.  Volume  I  examines  the  WBS  and 
Contractor  Cost  Data  Reporting  (CCDR)  system  with  a 
description  of  current  reporting  practices  and  implement- 
ation shortcomings.  Recommended  courses  of  action  to 
improve  reporting  requirements  and  thereby  improve  data 
quality  and  cost  estimates  are  proposed.  Major  cost  drivers 
for  AST&E,  from  both  the  perspective  of  Defense  Contractors 
and  Military  Flight  Test  Centers,  are  discussed.  Beginning 
in  Volume  II,  a  relational  data  base  system  is  introduced  to 
more  easily  evaluate  AST&E  cost  elements  and  physical/ 
performance  characteristics.  A  Contractor  Flight  Test  cost 
estimating  relationship  (CER)  is  developed  through  step-wise 
multiple  regression  analysis  of  data  gathered  from  Defense 
Contractors  and  Naval  Air  Systems  Command  (NAVAIR) . 
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I.  INTRODUCTION 

A.   PURPOSE  OF  PROJECT 

There  is  an  increasing  requirement  from  high  levels 
within  the  Government  that  the  Navy's  aircraft  cost  es- 
timators and  analysts  provide  explicit  estimates  for  the 
sub-elements  of  aircraft  system  test  and  evaluation  efforts. 
These  requests  ask  for  additional  data,  which  represent 
lower  levels  in  the  Aircraft  Work  Breakdown  Structure  (WBS) 
than  previously  required,  and  the  methods  necessary  to 
produce  them  are  needed  in  order  to  estimate  Aircraft  System 
Test  and  Evaluation  costs  in  greater  detail  and  with  better 
accuracy  than  is  currently  feasible. 

The  ultimate  objective  of  the  Naval  Air  Systems  Command 
(NAVAIR)  is  to  develop  cost  estimating  relationships  for  all 
of  the  following  System  Test  and  Evaluation  elements: 

1.  Wind  Tunnel  Article  and  Test 

2.  Static  Article  and  Test 

3.  Fatigue  Article  and  Test 

4.  Drop  Article  and  Test 

5.  Contractor  Flight  Tests 

6.  Flight  Test  Instrumentation 

7.  Miscellaneous  Ground  Tests 

8 .  Support  of  Contractor  Flight  Tests 


9.   Navy  Technical  Evaluation 

10.  Operational  Evaluation 

11.  Contractor  Support  of  Navy  Test. 

The  necessary  tasks  to  attain  this  objective  are  to: 

1.  Identify  physical,  performance,  and  programmatic 
parameters  which  are  the  primary  drivers  of  each  of 
the  major  functional  cost  categories  (engineering 
labor,  manufacturing  labor,  quality  control  labor,  and 
tooling  labor)  for  each  of  the  major  elements  of 
Aircraft  System  Test  and  Evaluation 

2.  Develop  parametric  cost  estimation  relationships  for 
each  of  the  above  System  Test  and  Evaluation  elements 
and  each  appropriate  functional  cost  category  by: 

a.  Defining  a  sample  of  pertinent  aircraft 
programs 

b.  Formulating  a  work  breakdown  structure/ 
functional  cost  element  matrix 

c.  Acquiring  historical  cost  data 

d.  Organizing  the  data  for  analysis 

e.  Employing  statistical  methods  to  develop  cost 
estimation  relationships,  and 

f.  Documenting  data,  sources,  rationale,  and 
methodology. 

Given  the  time  constraints,  attempting  to  concentrate 

on  all  eleven  elements  simultaneously  would  prove  to  be  both 

unmanageable  and  inefficient.   Therefore,  the  project 

sponsor  has  directed  that  the  initial  research  focus  on  the 

area  of  Contractor  Test  Flights.   If  conditions  permit.  Wind 

Tunnel  Tests,  Static  Tests,  and  Fatigue  Tests,  could  be 

included . 
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B.  IMPORTANCE  OF  THIS  STUDY 

The  importance  of  this  study  is  justified  by  the 
following  reasons.  First,  due  to  the  rapidly  increasing 
cost  of  new  aircraft  systems  in  a  budgetarily  constrained 
environment,  the  accuracy  of  cost  analysis  performed  by 
NAVAIR  becomes  imperative.  Second,  an  integral  study  of  the 
cost  problem  would  provide  more  insight  to  NAVAIR  to  answer 
the  ever  increasing  informational  needs  that  originate  as 
high  as  Congressional  Budget  Committees  and  Executive  Branch 
Agencies.  More  important,  it  is  crucial  for  cost  analysts 
to  take  into  consideration  new  cost  drivers,  particularly  in 
the  area  of  avionics.  Also,  it  is  felt  that  better  coor- 
dination with  contractors  via  an  improved  implementation  of 
the  Contractor  Cost  Data  Reporting  (CCDR)  system,  WBS 
format,  and  cost  estimation  model  will  result  in  the 
standardization  of  reporting  procedures  across  the  industry 
and  will  increase  the  likelihood  of  contractor  delivery  of 
guaranteed  performances  for  allocated  funds. 

C.  ORGANIZATION  OF  THIS  STUDY 

This  is  a  two  volume  research  study.  Volume  I  is 
organized  as  follows:  Chapter  II  provides  an  overview  of 
the  Contractor  Cost  Data  Reporting  (CCDR)  system  and  the 
Military  Standard  Work  Breakdown  Structure  (WBS).  A 
critical  analysis  of  the  current  cost  estimation  system  and 
its  shortcomings  is  the  focus  of  Chapter  III.   It  proposes  a 
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set  of  feasible  courses  of  action  that  could  be  implemented 
to  improve  the  present  system.  Chapter  IV  outlines  the 
various  steps  taken  to  accomplish  the  proposed  research. 
Specifically,  a  three-step  approach  is  undertaken: 
exploration,  analysis,  and  refinement.  Such  a  study  helps 
identify  research  processes  and  keys  upon  which  this  study 
should  concentrate.  These  issues  are  analyzed  in  detail  in 
Chapter  V.  Results  obtained  from  an  extensive  survey  of 
experts  in  the  contractors'  System  Test  and  Evaluation  field 
are  reported.  In  particular,  major  cost  drivers  that  are 
frequently  identified  include  aircraft  weight,  speed, 
avionics,  software,  and  management  practices  and  policies. 

Volume  II  focuses  on  the  analysis  of  data  discussed  in 
Volume  I,  Chapter  I  describes  the  available  current  data 
and  its  structure.  Current  parametric  techniques  are 
surveyed  in  Chapter  II.  A  requirements  analysis,  design, 
and  development  of  a  data  base  management  system  called 
TIGER  is  performed  in  Chapter  III.  The  purpose  of  the  data 
base  is  threefold.  First,  it  provides  immediate  and 
precise  answers  to  ad  hoc  queries  that  cost  analysts  could 
pose.  Second,  it  outlines  a  well-structured  basis  for 
standardizing  the  accounting  process  handled  by  various 
contractors.  Third,  it  can  be  used  to  assist  statistical 
analysis  for  cost  estimation.  The  data  collected  is 
analyzed,  discussed  and  cost  driver  models  are  developed  and 
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presented  in  Chapter  IV.   Finally,  conclusions  derived  as  a 
result  of  this  study  are  discussed  in  Chapter  V. 


13 


II.   DEPARTMENT  OF  DEFENSE  REQUIREMENTS 
FOR  CONTRACTOR  COST  DATA  REPORTING 


A.   THE  CONTRACTOR  COST  DATA  REPORTING  (CCDR)  SYSTEM 
1.   Introduction  to  CCDR 

During  the  decision  proce.ss  on  any  major  defense 
acquisition,  the  primary  focus  is  on  the  development  and 
attainment  of  performance  objectives.  An  accurate  acquisi- 
tion cost  estimate  is  an  equally  significant  program 
parameter  that  must  be  considered  in  detail  during  this 
decision  process.  Standardized,  accurate  and  detailed  cost 
data  are  indispensible  to  analysts  required  to  develop 
reliable  cost  estimates. 

In  July  1970,  the  Defense  Blue  Ribbon  Panel  Report 
stated,  "the  extent  of  availability  of  such  (cost)  data  in 
usable  form  is  a  limiting  factor  on  the  potential  accuracy 
of  cost  predictions."  ( NAVMAT  P-5241,  1973,  p.  i)  The 
Contractor  Cost  Data  Reporting  (CCDR)  system  was  established 
by  the  Office  of  the  Secretary  of  Defense  in  1973  to  provide 
for  "continual  improvement  in  the  ability  of  the  Department 
of  Defense  (DOD)  to  develop  and  use  valid  cost  estimates". 
(NAVMAT  P-5241,  1973,  p.  i)  This  system  is  intended  to 
provide  decision  makers  with  a  means  by  which  contract  costs 
and  other  related  data  can  be  collected  to  aid  in 
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acquisition  management.  The  reporting  requirements  are 
designed  to  collect  data  on  defense  weapon  systems  utilizing 
standard  data  definitions  and  reporting  structures  which 
facilitate  integration  with  other  defense  management 
systems.  To  provide  a  common  ground  from  which  to  view  this 
research,  the  following  excerpts  from  the  Contractor  Cost 
Data  Reporting  (CCDR)  System  ( NAVMAT  P-5241),  and  the 
Military  Standard  Work  Breakdown  Structures  for  Defense 
Materiel  Items  (MIL-STD-881A)  are  included  in  this  study. 
2 .      Purposes   of    the   CCDR   System 

The  data  collected  from  the  Defense  Contractors  is 
intended  to  be  used  by  DOD  components  in  establishing  cost 
estimating,  programming,  budgeting  and  procurement  respon- 
sibilities. This  data  collection  effort  is  to  provide  a 
common  data  base  and  assist  the  Department  of  Defense  in  the 
following   areas: 

a.  Preparing  estimates  in  support  of  the  Five  Year 
Defense   Program 

b.  Developing  independent  government  cost  estimates  in 
support  of  cost  and  price  analyses  and  contract 
negotiations 

c.  Evaluating   contractors'    proposals 

d.  Responding  to  requirements  for  summary  information  to 
the  Secretary  of  Defense  concerning  selected 
acquisitions  to  reflect  a  comparison  of  current 
estimates,  original  plans,  and  current  approved 
program   costs    and 

e.  Preparing  cost  estimates  for  major  system  review  by 
the  Defense  Systems  Acquisition  Review  Council  (DSARC) 
at  each  program  decision  milestone.  (NAVMAT  p-5241, 
1973,    p. 3-1) 
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3.   Reporting  Structure 

Reporting  requirements  are  differentiated  in 
accordance  with  the  following  contract  categories: 

a.  Category  I — Major  contracts  for  Prototypes  in  Advanced 
Development,  Full  Scale  Development,  and  Production, 
within  programs  which  are  estimated  in  the  Five  Year 
Defense  Program  to  require  a  cumlative  financing  for 
Research,  Development,  Test  and  Evaluation  in  excess 
of  $200  million  or  a  total  procurement  investment  in 
excess  of  $1  billion 

b.  Category  II — Contracts  for  defense  materiel  items  not 
satisfying  the  Category  I  criteria  but  selected  by  the 
DOD  component  for  cost  data  reporting  because  of 
complexity,  criticality,  future  procurement  plans  and 
contract  value.  (NAVMAT  P-5241,  1973,  p.  1-2) 

The  data  elements  which  produce  the  common  data  base 

are  generated  by  four  reports : 

1.  Cost  Data  Summary  Report  DD  Form  1921 

2.  Functional  Cost-Hour  Report  DD  Form  19  21-1 

3.  Progress  Curve  Report  DD  Form  1921-2 

4.  Plant-Wide  Data  Report  DD  Form  1921-3 
These  standardized  reports  are  designed  to  satisfy  a  wide 
range  of  weapon  system  acquisitions.  The  following  excerpts 
from  NAVMAT  P-5241  provide  a  basic  description  of  these  four 
CCDR  system  reports  which  are  submitted  at  varying  frequen- 
cies as  negotiated  in  the  contract: 

a.   DD  Form  19  21 — Cost  Data  Summary  Report 
(Figure  2-1) 

Primarily  designed  for  Category  I  contracts,  the 

Cost  Data  Summary  Report  summarizes  all  activities  included 

in  the  contract  and  aggregates   cost   against   the  reporting 
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elements  selected  from  the  work  breakdown  structures  defined 
in  MIL-STD-881  and  specified  in  the  contract.  WBS  elements 
below  Level  3  of  MIL-STD-881  may  be  designated  for  CCDR  but 
should  be  limited  to  those  for  which  cost  data  can  be 
realistically  utilized.  The  Cost  Data  Summary  Report  is 
also  used  to  present  the  contractor ' s  program  estimate  for 
RFP • s r  program  reviews,  or  special  studies  in  accordance 
with  the  fiscal  years  and  quantities  specified  by  the  DOD 
component  for  the  total  program.  (NAVMAT  P-5241,  1973,  p.  3- 
3) 

b.  DD  Form  1921-1 — Functional  Cost-Hour  Report 
(Figure  2-2) 

The  Functional  Cost-Hour  Report  is  the  means  of 
identifying  and  collecting  comparable  functional  costs, 
e.g.,  engieering,  tooling,  manufacturing,  for  (1)  specific 
contracts  and  (2)  estimates  for  the  fiscal  years  and 
quantities  specified  by  the  DOD  component  for  the  total 
program.  Reports  may  be  required  for  recurring,  non-recurr- 
ing, and  total  costs,  as  determined  and  specified  by  the  DOD 
contracting  component.  (NAVMAT  P-5241,  1973,  p.  3-9) 

c.  DD  Form  1921-2 — Progress  Curve  Report 
(Figure  2-3) 

The  Progress  Curve  Report  provides  a  unit  or  an 
average  unit  cost  of  the  unit  or  lot  accepted  during  the 
report  period.  All  costs  reported  on  this  form  are  recurr- 
ing, (NAVMAT  P-5241,  1973,  p.  3-17) 
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d.   DD  Form  1921-3 — Plant  Wide  Data  Report 

(Figure  2-4)  * 

The  report  shall  be  prepared  based  on  the 
contractor's  accounting  system  and  the  estimating  procedure. 
The  reporting  dates  should  be  established  to  coincide  with 
the  contractor's  fiscal  year.  It  will  be  submitted  for 
Category  I  contracts  only.  This  report  is  a  standardized 
plant-wide  overhead  report  which  replaces  the  various  types 
of  overhead  analyses  now  provided  to  Administrative  Con- 
tracting Officers  (ACOs)  for  major  acquisition.  (NAVMAT  P- 
5241,  1973,  p.  3-24) 

The  Contractor  Cost  Data  Reporting  system 
pamphlet,  NAVMAT  P-5241,  explains  how  to  complete  these 
forms  and  lists  definitions  of  cost  data  elements  to  include 
engineering,  tooling,  quality  control  and  manufacturing. 
4 .   Reporting  Elements 

The  reporting  elements  required  for  data  collection 
are  included  in  the  Request  for  Proposal  and/or  the  initial 
contract.  The  level  of  detail  to  be  included  in  the 
contractor  cost  reports  will  be  limited  to  that  which  can  be 
realistically  generated  by  the  contractor  and  utilized  by 
the  appropriate  DOD  contracting  component. 

Reporting  .elements  for  Category  I  contracts  are 
selected  from  the  Work  Breakdown  Structure  (WBS)  as  defined 
in  Military  Standard — Work  Breakdown  Structures  for  Defense 
Materiel  Items  (MIL-STD  881).  Reporting  elements  for 
Category  II  contracts  are  to  utilize  the  WBS  elements 
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whenever  possible  or  other  criteria  most  readily  reported  by 
the  contractor  using  their  existing  management  and  account- 
ing systems. 

5.   Direct  Labor  Hour  Definitions 

a.  Engineering  Direct  Labor  Hours 

The  hours  expended  in  the  study,  analysis, 
design, development ,  evaluation,  and  redesign  of  the 
specified  reporting  element.  Includes  the  prepara- 
tion of  specifications,  drawings,  parts  lists, 
wiring  diagrams,  technical  coordination  between 
engineering  and  manufacturing,  vendor  coordination, 
test  planning  and  scheduling,  analysis  of  test 
results,  data  reduction,  and  report  preparation. 
This  also  includes  the  determination  and 
specification  of  requirements  for  reliability, 
maintainability  and  quality  control.  (NAVMAT  P- 
5241,  1973,  p.  4-1) 

b.  Tooling  Direct  Labor  Hours 

The  hours  expended  in  the  planning,  design, 
fabrication,  assembly,  installation,  modification, 
maintenance  and  rework  of  all  tools,  including 
assembly  tools,  dies,  jigs,  fixtures,  master  forms, 
gauges,  handling  equipment,  load  bars,  work 
platforms  (including  installation  of  utilities 
thereon),  and  test  equipment  (such  as  checkers  and 
analyzers  in  support  of  manufacturing  the  specified 
reporting  element).  This  entry  includes  hours 
expended  in  the  determination  of  tool  requirements, 
planning  of  fabrication  and  assembly  operations, 
maintaining  tool  records,  establishing  make-or-buy 
plans  and  manufacturing  plans  on  components  and 
equipment,  scheduling  and  controlling  all  tool 
orders,  and  programming  and  preparation  of 
templates  and  patterns,  and  form  block  manufacture. 
(NAVMAT  P-5241,  1973,  p.  4-2) 

c.  Quality  Control  Direct  Labor  Hours 

The  hours  expended  in  the  design  and 
implementation  of  the  necessary  controls  to  ensure 
that  a  manufacturing  process  produces  an  item  or 
product  meeting  prescribed  standards.  Includes 
such  tasks  as  receiving  inspection,   in-process  and 
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final  inspection  of  tools,  parts,  subassemblies 
and  complete  assemblies,  and  reliability  testing 
and  failure  report  reviewing;  also  included  are 
such  tasks  as  the  establishment  of  acceptable 
quality  level  (AQL)  and  statistical  methods  for 
determining  performance  of  manufacturing  processes. 
The  preparation  of  reports  relating  to  these  tasks 
are  to  be  considered  quality  control  effort. 
(NAVMAT    P-5241,    1973,    p.    4-2) 

d.      Manufacturing   Direct   Labor   Hours 

The  hours  expended  on  or  chargeable  to  such 
operations  as  production  scheduling  and  expediting, 
fabrication,  processing,  subassembly,  final 
assembly,  reworking,  modification,  experimental 
production,  and  installation  of  parts  and 
equipment,  power  plants,  boosters,  electronic 
equipment,  explosives,  and  other  ordnance  items 
(including  government  furnished  equipment)  and  the 
proving  of  such  equipment,  instruments,  etc.,  for 
the  specified  reporting  element.  This  includes  the 
construction  of  detail  parts  from  raw  materials.  It 
includes  hours  expended  in  the  cutting,  forming, 
stretching  and  blanking  operations  performed  on 
material  of  any  kind  (metal,  wood,  plastic,  glass, 
cloth,  tubing,  etc.)  to  make  individual  parts.  It 
includes  bench  assemblies  of  all  detail  parts,  all 
minor  and  major  assemblies,  mating  or  jointing  of 
primary  sections,  installation  of  special  and 
general  equipment,  instruments  and  accessories 
performed  after  the  mating  and  all  other 
preparation  and/or  processing  including  all 
flashing  operations,  annealing,  heat  treating, 
baking,  refrigeration,  anodizing  plating,  painting 
and  dope  operations  and  preflight  and  production 
service  operations,  etc.  (NAVMAT  P-5241,  1973,  p. 
4-3) 

6.      Dollar  Values 

The     CCDR    system    provides     for    the    reporting    of 

various    categories    of    contractor    costs,    e.g..     Direct    Labor 

Dollars,       Overhead,       Material,       and      Other      Direct    Charges. 

However,        because       of    the       fact    that    all    of    the    data    for 

this   research  would   be   historical,    coupled  with   the    lack   of 
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constant  dollar  data  due  to  inflation,  it  would  be  statisti- 
cally more  significant  to  use  Direct  Labor  Hours  as  the  only 
cost  data  element  in  this  project. 

B.   WORK  BREAKDOWN  STRUCTURE 
1 .   Definition 

A  Work  Breakdown  Structure  (WBS)  is  a  product- 
oriented  family  tree  composed  of  hardware,  services,  and 
data  which  result  from  project  engineering  efforts  during 
the  development  and  production  of  a  defense  materiel  item 
and  which  completely  defines  the  pro ject /program.  A  WBS 
displays  and  defines  the  product ( s  )  to  be  developed  or 
produced  and  relates  the  elements  to  work  to  be  accomplished 
to  each  other  and  to  the  end  product.  (MIL-STD-881A,  1975, 
p.  2) 

When  a  weapons  system  is  viewed  as  a  whole,  the 
importance  of  standardized  reporting  elements  is  reduced. 
The  Level  1  total  system  hours/cost  will  remain  the  same 
regardless  of  what  specific  indentured  sub-element  they  are 
reported  against.  However,  as  individual  reporting  elements 
are  examined  at  each  level  of  the  Work  Breakdown  Structure 
the  need  for  these  standardized  definitions  is  imperative. 
A  lack  of  standardization  will  result  in  inconsistent  data, 
and  in  turn,  unreliable  cost  estimates. 
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2.      Purposes   of   the   Work   Breakdown    Structure   MIL-STD 
881A 

As    stated    in    the    MIL-STD-881A,    the    purpose    of    the 

WBS    is    to    establish    criteria    governing    the    preparation    and 

employment    of    work    breakdown    structures    for    use    during    the 

acquisition    of    designated    defense    materiel    items.        These 

work    breakdown    structures    would    provide    a    consistent    and 

visible    framework   that    facilitates: 

a.  A  more  effective  management  and  technical  base  for 
planning  and  assigning  management  and  technical 
responsibilities  by  operations  within  the  government 
offices  responsible  for  the  acquisition  of  defense 
materiel  items  and  those  contractors  furnishing  the 
items 

b.  More  consistent  control  over  and  reporting  of  the 
progress  and  status  of  engineering  and  other 
contractor  efforts,  resource  allocations/  cost 
estimates,  expenditures,  and  procurement  actions 
throughout   the   acquisition   of   defense   materiel   items 

c.  Consideration  of  total  life  cycle  effects,  including 
development,  production,  activation,  operational  use, 
and  phase-out,  when  making  system  development  and 
acquisition   decisions. 

The  uniformity  in  definition  and  approach  for 
developing  the  upper  three  levels  of  the  WBS  established  by 
this  standard  is  expected  to  assure  compatibility  of 
multiple-data  requirements.  The  benefits  expected  from 
increased  uniformity  in  the  generation  of  work  breakdown 
structures  and  their  application  to  management  practices 
will  be  realized  by  the  improved  interpretation  and  recon- 
ciliation of  all  reports  prepared  to  this  uniform  framework 
throughout  acquisition  of  a  defense  materiel  item.  (MIL-STD- 
881A,    1975,    p.    ii) 
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3.   Additional  Work  Breakdown  Structure  Definitions 

a.   Sununary  Work  Breakdown  Structure  (Summary  WBS) 

A  Summary  Work  Breakdown  Structure  consists  of  the 
upper  three  levels  of  a  WBS  prescribed  by  the  standard  and 
having  uniform  element  terminology,  definition,  and 
placement  in  the  family-tree  structure.  The  upper  three 
levels  of  a  summary  WBS  have  been  organized  within  the 
following  categories  of  defense  materiel  items; 

1.  Aircraft  Systems 

2.  Electronics  Systems 

3.  Missile  Systems 

4.  Ordnance  Systems 

5 .  Ship  Systems 

6 .  Space  Systems 

7.  Surface  Vehicle  Systems.  (MIL-STD-881A,  1975,  p.  2) 

The   three    levels   specified    are   defined   as 
follows: 

1.  Level  1  is  the  entire  defense  materiel  item;  for 
example,  the  Minuteman  ICBM  System,  the  LHA  Ship 
System,  or  the  M  109A1  Self-Propelled  Howitzer  System. 
Usually,  Level  1  is  directly  identified  in  the  DOD 
programming/budget  system  either  as  an  integral 
program  element  or  as  a  project  within  an  aggregated 
program  element 

2.  Level  2  elements  are  major  elements  of  the  defense 
materiel  item;  for  example,  a  ship,  an  air  vehicle,  a 
tracked  vehicle,  or  aggregations  of  services,  (e.g., 
systems  test  and  evaluation);  and  data 

3.  Level  3  elements  are  those  subordinate  to  Level  2;  for 
example,  an  electric  plant,  an  airframe,  the  power 
package/drive   train,   or   type   of   service,   (e.g., 


30 


development  test  and  evaluation);  or  item  of  data 
(e.g.,  technical  publications). 

b.  Project  Summary  Work  Breakdown  Structure 
(Project  Summary  WBS ) 

Project  Summary  WBS  is  a  Summary  WBS  tailored  to 

a  specific  defense  materiel  item. 

c.  Contract  Work  Breakdown  Structure  (Contract  WBS) 
A  Contract  WBS  is  defined  as  the  complete  WBS 

for  a  contract,  developed  and  used  by  a  contractor  in 
accordance  with  this  standard  and  the  contract  work 
statement. 

d.  Project  Work  Breakdown  Structure  (Project  WBS) 
A  Project  WBS  is  defined  as  the  complete  WBS  for 

the  project,  containing  all  WBS  elements,  related  to  the 
developments  and/or  production  of  the  defense  materiel  item. 

e.  Work  Breakdown  Structure  Element 

A  work  breakdown  structure  element  is  a  discrete 
portion  of  a  work  breakdown  structure.  A  WBS  element  may  be 
either  an  identifiable  item  of  hardware,  set  of  data,  or  a 
service.  (MIL-STD-881A,  1975,  p.  3) 

C.   SUMMARY  WORK  BREAKDOWN  STRUCTURE  AND  DEFINITIONS  OF 
AIRCRAFT  SYSTEMS 

1 .   Project  Scope 

As  previously  discussed,  MIL-STD-881A  provides  seven 

Summary  WBS ' s  for  use  by  all  contractors  and  DOD  components 

in  the  development  of  work  breakdown  structures  for  the 

acquisition  of  defense  materiel  systems.   Since  this  study 
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addresses  only  aircraft  systems,  the  WBS  in  Appendix  A  of 
MIL-STD-881A  will  be  the  only  structure  presented  and 
discussed. 

2 .   Summary  Work  Breakdown  Structure 

The  following  is  a  Summary  Work  Breakdown  Structure 
for  an  aircraft  system  (MIL-STD-881A,  1975,  p.  19)  with  the 
elements  of  System  Test  and  Evaluation  in  bold  type: 

Level  1  Level  2         Level  3 


Aircraft  system 

Air  vehicle 


Training 


Airframe 
Propulsion  unit 
Other  propulsion 
Communications 
Navigation/guidance 
Fire  control 
Penetration  aids 
Reconnaissance  equipment 
Automatic  flight  control 
Central  integrated  checkout 
Antisubmarine  warfare 
Auxiliary  electronics 

equipment 
Armament 

Weapons  delivery  equipment 
Auxiliary  armament/weapons 

delivery  equipment 

Equipment 

Services 

Facilities 


Peculiar  support 
equipment 

Organizational/intermediate 
(Including  equipment 
common  to  depot ) 
Depot  (Only) 
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Level  1 


Level  2 


Level  3 


System  test  and 
evaluation 


System/pro  ject 
management 


Development  test  and 

evaluation 
Operational  test  and 

evaluation 
Mockups 

Test  and  evaluation  support 
Test  facilities 


System  engineering 
Project  management 


Data 


Technical  publications 
Engineering  data 
Management  data 
Support  data 
Data  depository 

Operational/site 
activation 

Contractor  technical 

support 
Site  construction 
Site/ship/vehicle 
conversion 

Common  support  equipment 

Organizational/intermediate 
(Including  equipment 
common  to  depot ) 
Depot  (Only) 

Industrial  facilities 

Construction/conversion/ 

expansion 
Equipment  acquisition  or 

modernization 
Maintenance 

Initial  spares  and 
initial  repair  parts 

(Specify  by  allowance  list, 
grouping  or  hardware 
element) 
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3.   System  Test  and  Evaluation  Definitions 

a.  System  Test  and  Evaluation 

The  System  Test  and  Evaluation  element  refers  to 
the  use  of  prototype,  production,  or  specially  fabricated 
hardware  to  obtain  or  validate  engineering  data  on  the 
performance  of  the  aircraft  system.  This  element  includes 
the  detailed  planning,  conduct,  support,  data  reduction  and 
reports  from  such  testing,  and  all  hardware  items  which  are 
consumed  or  planned  to  be  consumed  in  the  conduct  of  such 
testing.  It  also  includes  all  effort  associated  with  the 
design  and  production  of  models,  specimens,  fixtures,  and 
instrumentation  in  support  of  the  test  program.  Test 
articles  which  are  complete  units  (i.e.,  functionally 
configured  as  required  by  the  aircraft  equipment)  are 
excluded.  Development  component  acceptance,  etc.,  testing 
which  can  be  specifically  associated  with  the  hardware 
element,  unless  these  tests  are  of  special  contractual  or 
engineering  significance  (e.g.,  associate  contractor),  are 
also  excluded.  (MIL-STD-881A,  1975,  p.  24) 

b.  Development  Test  and  Evaluation 

The  Development  Test  and  Evaluation  (DT&E) 
element  refers  to  that  test  and  evaluation  conducted  to: 

1.  Demonstrate  that  the  engineering  design  and  develop- 
ment process  is  complete 

2.  Demonstrate  that  the  design  risks  have  been  minimized 

3.  Demonstrate  that  the  system  will  meet  specifications 
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4.  Estimate  the  system's  military  utility  when  introduced 

5.  Determine  whether  the  engineering  design  is  support- 
able (practical,  maintainable,  safe,  etc.),  for 
operational  use 

6.  Provide  test  data  with  which  to  examine  and  evaluate 
tradeoffs  against  specification  requirements,  life 
cycle  cost,  and  schedule. 

DT&E  is  planned,  conducted,  and  monitored  by  the 
developing  agency  of  the  DOD  component.  It  includes,  for 
example,  such  models  and  tests  as  wind  tunnel,  static,  drop, 
and  fatigue;  integration  ground  tests,  engine  military 
qualification  tests  (MQT),  preliminary  flight  rating  tests 
(PFRT),  test  bed  aircraft  and  associated  support; 
development  flight  test,  test  instrumentation,  test 
equipment  (including  its  support  equipment),  chase  aircraft 
and  support  thereto,  etc.  (MIL-STD-881A,  1975,  p.  24) 

c.   Operational  Test  and  Evaluation 

The  Operational  Test  and  Evaluation  element 
refers  to  that  test  and  evaluation  conducted  by  agencies 
other  than  the  developing  command  to  assess  the  prospective 
systems'  military  utility,  operational  effectiveness, 
operational  suitability,  logistics  supportability  (including 
compatibility,  interoperability,  reliability,  maintainabili- 
ty, logistic  requirements,  etc.),  cost  of  ownership,  and 
need  for  any  modifications.  Initial  Operational  Test  and 
Evaluation  (lOT&E)  conducted  during  the  development  of  a 
weapon  system  will  be  included  in  this  element.    This 
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element  encompasses  such  tests  as  flight  tests,  sea  trials, 
etc.,  and  support  thereto,  required  to  prove  the  operational 
capability  of  the  deliverable  system.  It  also  includes 
contractor  support  (e.g.,  technical  assistance,  maintenance, 
labor,  material,  etc.)  consumed  during  this  phase  of 
testing.     (MIL-STD-881A,    1975,    p.    24) 

d.  Mockups 

The  Mockups  element  refers  to  the  design 
engineering  and  production  of  system  or  subsystem  mockups 
which  have  special  contractual  or  engineering  significance, 
or  which  are  not  required  solely  for  the  conduct  of  one  of 
the  above  elements  of  testing.  (MIL-STD-881A,  1975,  p.  24) 

e.  Test  and  Evaluation  Support 

The  Test  and  Evaluation  Support  element  refers 
to  all  support  elements  necessary  to  operate  and  maintain 
systems  and  subsystems  dxiring  flight  test  and  evaluation 
which  are  not  consumed  during  the  flight-testing  phase  and 
other  support  requirements  that  are  not  allocable  to  a 
specific  phase  of  testing.  This  element  includes,  for 
example,  repairable  spares,  repair  of  repairables,  repair 
parts,  contractor  technical  support,  etc.,  not  allocable  to 
preceding  test  and  evaluation  elements.  Operational  and 
maintenance  personnel,  consumables,  special  fixtures, 
special  instrumentation,  etc.,  which  are  utilized  and/  or 
consumed  in  a  single  element  of  testing  and  which  should. 
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therefore,  be  included  under  that  element  of  testing  are 
excluded.  (MIL-STD-881A,  1975,  p.  25) 
f.   Test  Facilities 

The  Test  Facilities  element  refers  to  those 
special  test  facilities  required  for  performance  of  the 
various  developmental  tests  necessary  to  prove  the  design 
and  reliability  of  the  system  or  subsystem.  This  element 
includes  for  example,  engine  test  fixtures,  white  rooms, 
test  chambers,  etc.  The  brick-and-mortar-type  facilities 
allocable  to  industrial  facilities  are  excluded.  (MIL-STD- 
881A,  1975,  p.  25) 
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III.   COST  ANALYSIS  PROBLEMS  IN  TEST 
AND  EVALUATION  OF  AIRCRAFT 

A.   INTRODUCTION 

The  Cost  Analysis  Division  of  the  Naval  Air  Systems 
Command  (NAVAIR)  in  the  early  stage  of  concept  formulation 
initiates  action  to  estimate  costs  and  programmatics.  These 
concepts  begin  prior  to  the  issuance  of  the  initial  request 
for  proposals  when  there  is  a  perceived  need  for  a  new 
aircraft  system  to  the  end  of  the  demonstration/validation 
phase  of  the  aircraft  system  development  life  cycle. 

The  current  practices  in  data  collection  and  standar- 
dization have  been  primarily  designed  for  cost  analysis  and 
estimation.  There  are  several  reasons  for  this  lack  of 
analytical  capability.  First,  definitions  of  costs  elements 
are  sometimes  ambiguous.  Too  frequently,  there  is  a  lack  of 
mutual  effort  to  eliminate  this  ambiguity,  which  often  leads 
to  a  lack  of  standardization  within  contractors  '  reports 
required  by  the  Department  of  Defense  (DOD).  In  addition  to 
their  ambiguity,  MIL-STD-881A  definitions  apply  only  to 
Levels  1,  2,  and  3  of  the  Work  Breakdown  Structure  (WBS). 
Below  Level  3,  definitions  are  generally  tailored  to  the 
specific  contract.  This  lack  of  specificity,  for  whatever 
reason,  makes  it  difficult  to  normalize  data  submitted  under 
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different  contracts,  which  results  in  the  inability  to 
effectively  support  trade-off  analysis  in  the  cost  estimat- 
ing process.  NAVAIR  established  a  CCDR  Committee  to  enforce 
compliance  and  provide  oversight  to  ensure  consistency 
across  programs,  where  appropriate.  Figure  3.1  reproduces 
the  current  draft  generic  structure  of  the  WBS  used  by 
NAVAIR  when  establishing  the  Project  Summary  WBS  for  use  in 
estimating/negotiating  individual  aircraft  programs.  The 
final  decision  as  to  whether  or  not  this  WBS  is  used  in  the 
negotiation  of  a  program  rests  with  the  Program  Manager. 

B.   THE  COST  ANALYSIS  PROCESS 

1 .   Description  of  Current  Practices 

a.   Development  of  the  Project  Work  Breakdown 
Structure 

The  cost  analysis  process  is  triggered  at  the 

beginning  of  the  conceptual  phase  of  the  Defense  Material 

Acquisition  process.   A  Project  Summary  Work  Breakdown 

structure  is  developed  by  the  DOD  component  utilizing  the 

category  summaries  in  the  appendices  of  MIL-STD-881A.   This 

Project  Summary  WBS  will  be  identified  to  the  concerned 

contractors  during  RFP  or  solicitation.   This  proposed 

structure  will  be  negotiated  with  the  contractors,  who  may 

recommend  changes  to  the  proposed  Contractor  WBS  that  they 

believe  would  improve  its  effectiveness  in  achieving  the 

goals  of  the  system  acquisition.   The  negotiated  Contract 

WBS  will  then  be  extended   as  far  below  Level  3  as  required 
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Level 

v;bs  #  L   1   1   i   1   i 

2000  Systems  Test  &  Evaluation  (ST&E) 

2100  Development  Test  &  Evaluation 

2110  Contractor  Flight  Tests 

2111  Instrumentation 

2112  Functional  Ground  Checkout 

2113  Flight  Test  &  Demonstration 

2114  Flight  Test  Support 

2114.1  Flight  Test  Spares  &  Repair 

Parts 

2114.2  Support  Equipment 

2114.3  Contractor  Technical/ 

Maintenance  Services 

2120  Wind  Tunnel  Article  &  Test 

2121  Wind  Tunnel  Article 

2122  Wind  Tunnel  Test 

2130  Static  Article  &  Test 

2131  Static  Article 

2132  Static  Test 

2140  Fatigue  Article  &  Test 

2141  Fatigue  Article 

2142  Fatigue  Test 


Figure  3-lA.   AIR-524  Standard  Aircraft  CCDR  Work 
Breakdown  Structure  Format 
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Level 

WBS  #  L    i.    1    1    1    i 

2150  Drop  Article  &  Test  ■ 

2151  Drop  Article 

2152  Drop  Test 
2160  Simulation  Testing 

2170  Avionics  Integration  Testing 

2171  Test  Bench/Laboratory 

2172  Flying  Test  Bed 

2173  Air  Vehicle  Equipment 

2174  Avionics  Test  Program 

2175  Software 

2180  Navy  Flight  Test  Prelim  DT  II 

2190  Navy  Flight  Test  Final  DT  II 

21A0  Navy  Flight  Test  DT  III 

21_0  Other  DT&E  (Specify  at  Level  4) 

2200  Operational  Test  &  Evaluation 

2210  Preliminary  OT  II 

2220  Final  OT  II 

2230  OT  III  &  IV 

2300  Mockups 

2400  Test  &  Evaluation  Support 

2410  Test  &  Evaluation  Spares  &  Repair 

Parts 

2420  Test  &  Evaluation  Support  Equipment 

2430  Test  St  Evaluation  Technical/  . 

Maintenance  Services 

2500  Test  Facilities 

2510  Avionics  Integration  Facility 

25_0  Other  (Specify  at  Level  4) 

2600  Other  ST&E  (Specify) 

Figure  3-lB.   AIR-524  Standard  Aircraft  CCDR  Work 
Breakdown  Structure  Format 
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to  clearly  define  the  extent  of  the  contract.  The  Contract 
WBS  is  combined  with  the  Project  Summary  WBS  to  form  the 
Project  WBS. 

b.  Purpose  of  the  Project  Work  Breakdown  Structure 
The  Project  Summary  WBS,  and  its  derivatives, 

are  established  early  in  the  acquisition  process  to  provide 
a  managerial  and  technical  framework  for  all  activities 
throughout  the  acquisition  life  cycle.  DOD  Project  Managers 
are  directed  to  utilize  these  work  breakdown  structures, 
"as  a  coordinating  medium  in  planning  for  further  systems 
engineering,  resource  allocation,  cost  estimates ,  contract 
actions,  and  work  execution.  The  reporting  of  progress, 
performance,  and  engineering  evaluations,  as  well  as 
financial  data,  shall  be  based  on  the  Project  WBS".  (MIL- 
STD-881A,  1975,  p.  5) 

c.  Initial  Cost  Estimates 

An  initial  part  of  the  acquisition  process 
involves  the  Cost  Analysis  Division  of  NAVAIR.  This 
division  is  responsible  for  providing  an  initial  rough  order 
of  magnitude  cost  estimation  of  the  proposed  system,  which 
will  be  continually  refined  toward  absolute  accuracy 
throughout  the  program  life  cycle.  Once  a  program  is 
initiated,  the  Cost  Analysis  Division,  based  on  system  type 
and  acquisition  phase,  assigns  analysts  to  perform  more 
detailed  cost  estimating  and  establish  the  initial  cost 
reporting  requirements.   "During  preparation  of  the  Request 
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for  Proposals  (RFP)  or  solicitation,  the  procuring  activity, 
should  determine  the  CCDR  requirements.  This  involves 
assessment  of  cost  estimating  needs  and  the  contents  of  the 
data  elements  which  can  be  generated  with  the  four  reporting 
forms".  (NAVMAT  P-5241,  1973,  p.  2-1)  The  initial  task 
reports  are  further  refined  by  a  selected  CCDR  steering 
committee  composed  of  cost  experts.  The  Program  Analysis 
Evaluation  ( PAE )  section  of  the  Office  of  the  Secretary  of 
Defense  (OSD)  reviews,  directs  modifications  if  necessary, 
and  approves  the  CCDR. 

d.   Reporting  Requirements 

With  some  exceptions,  the  data  from  contractors, 
normally  down  to  Level  3,  are  submitted  to  the  NAVAIR  Cost 
Division  every  six  months  for  dissemination  and  validation 
for  internal  use.  NAVAIR  then  systematically  distributes 
all  the  CCDR  reports  to  pertinent  parties  within  DOD  (e.g., 
OSD,  ASN,  PAE).  This  procedure  has  been  continuously  used 
since  1966  and  there  is  no  indication  that  this  routine  will 
be  altered  in  the  foreseeable  future  due  to  two  main 
reasons.  First,  the  current  historical  data  base  is  seldom 
implemented  below  Level  3  of  the  WBS ,  OSD  probably  feels  the 
need  to  maintain  the  integrity  of  this  data  base.  Second, 
the  Office  of  Management  and  Budget  (0MB)  has  established 
guidelines  requiring  OSD  to  minimize  the  reporting  require- 
ments placed  on  the  contractors  by  the  CCDR  system.   This 
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seems  to  concur  with  the  administration  policy  to  reduce  the 

volume  of  bureaucratic  paperwork. 

2.   Implementation  Shortcomings  of  the  Present  System;  A 
Cost  Analysis  Perspective 

The  current  system  requires  contractors  to  report  at 
varied  WBS  levels,  however,  from  a  cost  estimating  perspec- 
tive reporting  requirements  continue  to  inappropriately 
fluctuate  from  one  contract  to  another.  Regardless  of  the 
depth  required  in  the  WBS,  contractors  claim  (despite  the 
fact  that  the  CIR/CCDR  system  reporting  requirements  have 
been  known  since  the  mid  60 's)  that  it  is  too  difficult  to 
convert  their  accounting  systems  to  accommodate  the  defini- 
tions given  by  the  CCDR  system;  an  argument  with  which 
NAVAIR  strongly  disagrees.  To  further  complicate  the 
problem,  dual  source  and  sub-contracts  are  often  granted. 
Due  to  their  competitiveness,  contractors  often  cannot,  or 
even  do  not  want  to  cooperate  with  other  contractors . 

For  a  long  time,  by  analyzing  all  of  the  elements  of 
the  work  breakdown  structure,  discrete  answers  were  genera- 
ted to  reply  to  trade-off  questions  coming  from  high-level 
authorities  (e.g.,  ASN,  SYSCOM) .  However,  the  required  data 
are  often  not  available.  When  they  are  available,  their 
quality  is  questionable,  making  it  difficult  to  provide 
reliable  estimates  for  lower  level  WBS  elements.  Usually, 
answers  to  questions  require  data  below  Level  3  of  the  WBS. 
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It  is  difficult  to  obtain  periodic  (six-month 
period)  reporting  costs  below  Level  3  of  the  WBS  without 
encountering  resistance  from  OSD  and  contractors.  The 
latter  often  claim  that  it  would  cost  too  much  to  maintain 
such  a  data  base.  This  argument  may  be  true  for  some 
contractors,  but  it  is  not  true  industry-wide.  Interviews 
performed  for  this  study  have  indicated  that  most  contrac- 
tors have  for  a  long  time  maintained  data  bases  with  monthly 
data  for  internal  use  that  potentially  contain  information 
for  detailed  and  frequent  estimation  needs.  Most  of  the 
contractors  interviewed  maintain  data  at  least  one  level 
below  that  required  by  the  Project  WBS.  In  fact,  some  of 
these  data  bases  reflect  elements  down  to  Levels  8  and  9. 
It  would  probably  be  neither  as  expensive  nor  as  time- 
consuming/  as  contractors  claim,  to  provide  periodic  cost 
reports  below  Level  3,  especially  considering  the  fact  that 
CIR/CCDR  requirements  have  existed  since  1967,  and  the 
capabilities  of  modern  day  computers  to  ref ormat/reaggregate 
contractor's  cost  accounts  into  CCDR  format. 

C.   RECOMMENDED  COURSE  OF  ACTION 

1 .   Necessity  to  Break  the  WBS  to  Lower  Level 

Regardless  of  fiscal,  political  and  technical 
considerations,  it  is  critical  for  the  cost  analyst  to  have 
access  to  accurate  data  below  Level  3  of  the  WBS  for 
meaningful  data  analysis.   In  Level  2,     System  Test  and 
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Evaluation  WBS  and  distribution  of  Direct  Contract  Labor 
Hours  of  a  typical  military  fixed-wing  aircraft.  Of  the 
four  Level  3  sub-elements,  Development  Test  and  Evaluation 
accounts  for  93.6%  of  the  contract  total  System  Test  and 
Evaluation  labor  hours. 

The  Level  3  element  Development  Test  and  Evaluation 
(WBS  #  2100)  had  to  be  broken  down  to  its  Level  4  sub- 
elements  to  capture  its  major  cost  WBS  elements.  By 
referring  to  WBS  #  2119,  it  can  be  found  that  Contractor 
Flight  Tests  account  for  43.3%  of  Level  3  Development  Test 
and  Evaluation,  which  corresponds  to  40.6%  of  Level  2  System 
Test  and  Evaluation.  The  three  remaining  Level  3  sub- 
elements  account  for  only  6.4%  of  the  total  contract  labor 
hours.  In  particular,  TECHEVAL  costs  which  are  on  Level  3 
(WBS  #  2200)  were  minimal,  amounting  to  0.4%  of  Level  2 
System  Test  and  Evaluation. 

In  the  long  run,  it  would  be  advisable  to  restruc- 
ture the  WBS  system  by  revising  the  hierarchy  of  the  WBS 
elements.  One  recommendation  for  revision  would  be  to 
remove  Development  Test  and  Evaluation  (WBS  #  2100)  from 
Level  3  and  replace  it  with  selected  major  WBS  elements 
currently  located  on  Level  4.  In  addition  to  the  obvious 
choice  of  selecting  the  Contractor  Flight  Test  element  to  be 
moved   to   Level   3,   other   Level   4   sub-elements   contain 
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TABLE  3-1.   SYSTEM  TEST  AND  EVALUATION  DIRECT 
CONTRACT  LABOR  HOUR  DISTRIBUTION 


-Level Percentage  of  Total 

WBS  #234  Contract  Labor  Hours 


2000 

SYSTEM  T&E 

100 

2100 

DEV 

TEST 

93. 

.6 

2110 

SYSTEM  REQ 

9.4 

2111 

WIND  TUNNEL 

4.3 

2112 

STATIC  T&A 

5.8 

2113 

FATIGUE  T&A 

6.0 

2114 

A/V  SUB  TEST 

8.6 

2116 

AVNX  INT  TEST 

6.6 

2117 

ARM/WPN  INT 

0.7 

2118 

FLT  SIM  PROG 

1.5 

2119 

CTR  FLT  TEST 

40.6 

2190 

MISC 

6.6 

2191 

DROP  &  ACL  LOADS 

3.5 

2200 

TECHEVAL 

0. 

.4 

2220 

FLT  TEST  PRO  SPT 

0.4 

2400 

MOCKUPS 

4 

.5 

2500 

T&E 

SUPPORT 

1 

.5 
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significant  percentages  of  the  total  System  Test  and 
Evaluation  efforts  and  should  also  be  considered  for  a  move 
to  Level  3  of  the  WBS. 

The  prospective  of  implementing  a  revised  version  of 
the  WBS  system  depends  much  on  the  personalities  of  the  con- 
stituencies involved  in  the  process,  and  the  organizations 
created  around  programs.  System  Commands  should  ensure  that 
CCDR  oversight  committees  are  formed  to  review  current 
procedures,  and  enforce  proper  implementation.  The  coopera- 
tion of  contractors,  as  well  as  DOD  components,  will  be  a 
vital  part  of  this  effort,  which  will  involve  optimizing  and 
sharing  by  all  concerned, 

2.   Necessity  to  Provide  Time-Phased  Data  Reporting 

Data  should  be  reported  in  more  depth,  systematical- 
ly, and  in  a  time-phased  fashion.  Time  is  a  big  cost 
driver  in  some  elements.  For  example.  Contractor  Test 
Flight  costs  seem  to  relate  to  the  length  of  the  test 
periods.  A  side  benefit  of  this  would  be  that  cost  esti- 
mators would  have  an  improved  historical  base  for  profiling 
cost  estimates  into  annual  budget  increments  to  predict 
Research  and  Development  costs  for  each  year.  While  it  is 
difficult  to  derive  an  optimal  frequency  of  time  series  data 
reporting,  it  would  make  sense  to  argue  that  yearly  reports 
are  minimally  adequate.  Monthly  data  would  be  preferable 
since  they  would  capture  the  detailed  trend  of  labor  hours 
incurred  throughout  the  system  development  life-cycle,  but 
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might  not  be  cost  effective.   It  is  a  question  of  the  effort 

required  to  convert  data  from  contractor  to  the  CCDR  system 

format,  and  the  normalization  of  data  across  various 

contractor  inputs.   As  a  compromise,  semi-annual  data  would 

be  acceptable  and  more  cost  effective,  since  all  contractors 

have  computerized  accounting  systems  to  maintain  their  costs 

internally.   Due  to  the  long  process  of  aircraft  system 

development,  semi-annual  data  are  expected  to  provide 

sufficiently  detailed  information  for  trend  analysis. 

3 .   Implementation  of  a  well-defined  CCDR  Data  base 
System 

As  it  is  defined  in  Chapter  II,  the  purpose  of  the 
CCDR  system  is  to  provide  the  primary  data  base  for  use  in 
most  cost  estimating  efforts  including  procurement  manage- 
ment activities.  As  discussed  earlier,  the  realization  of 
this  effort  has  been  dampened  by  the  lack  of  standardized 
data  format  among  similar  programs  below  Level  3  of  the  WBS. 
It  seems  thus  evident  that  one  of  the  first  steps  to  be 
taken  is  to  elaborate  a  comprehensive,  consistent,  and 
precise  WBS  Elements  Dictionary  that  can  be: 

1.  Agreed  upon  by  all  DOD  components  as  well  as  the 
Defense  Industry 

2.  Implemented  for  cost  analysis  purposes. 

Ideally,  the  new  definitions  would  serve  as  foundations  for 
building  a  data  base  system  that  compiles  all  data  in  an 
accessible,  dynamic,  and  evolving  data  base,  readily  avai- 
lable for  retrieval  and  modification  for  modelling  and 
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analysis.  Historically,  required  reporting  elements  have 
been  inconsistently  determined  across  contracts  for  similar 
weapon  systems.  The  required  levels  of  reporting  within  the 
WBS  have  often  been  inconsistent  among  the  different  DOD 
components.  Individual  contracts  have  also  been  negotiated 
to  allow  some  obscuring  of  information  due  to  subcontractor 
reporting  variances  and  deficiencies.  Labor  cost  com- 
parisons in  terms  of  dollar  expenditures  do  not  provide  a 
basis  for  consistent  comparison  due  to  changing  economic 
price  levels.  Variances  between  contractor  methods  of 
accounting  also  generate  ambiguity.  For  instance,  some 
contractors  may  allocate  labor  hour  expenditures  for  direct 
maintenance  and  operational  support  of  the  test  vehicle  to 
engineering  costs,  while  others  may  allocate  the  same 
function  to  manufacturing  costs.  These  factors  do  not  imply 
inaccuracy  of  the  data,  but  do  induce  inconsistency  when 
comparing  and  correlating  individual  models  of  aircraft  and 
Defense  Contractors. 

The  utilization  of  the  Contractor  Cost  Data 
Reporting  system  data  base  to  provide  explicit  estimates  for 
sub-elements  of  aircraft  system  test  and  evaluation  has  not 
been  expedient.  Lower  levels  of  data  than  previously  found 
in  the  Work  Breakdown  Structure  should  be  analyzed  in  order 
to  effectively  obtain  the  desired  information  and  accurate  results. 
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IV.   RESEARCH  METHODOLOGY 

A.   OBJECTIVES  AND  SCOPE  OF  RESEARCH 

1 .  Objectives 

This  research  seeks  to:  (i)  assess  current  problems 
in  Cost  Analysis  in  Aircraft  System  Test  and  Evaluation, 
(ii)  establish  guidelines  for  estimating  data,  (iii)  design 
and  implement  a  database  structure  for  Aircraft  System  Test 
and  Evaluation,  and  (iv)  develop  parametric  cost  estimating 
relationships  for  selected  System  Test  and  Evaluation 
elements  and  four  functional  cost  categories.  Specifically, 
this  research  will  focus  on: 

1.  Defining  a  sample  of  pertinent  aircraft  programs 

2.  Formulating  a  WBS  structure/functional  cost  element 
matrix 

3.  Acquiring  historical  cost  data 

4.  Organizing  the  data  for  analysis 

5.  Employing  statistical  methods  to  develop  cost  estimat- 
ing relationships 

6.  Documenting  data,  sources,  rationale,  and  methodology. 

2 .  Scope 

This  research  attempts  to  apply  econometric  theory 
and  cost  analysis  in  the  area  of  Aircraft  Systems  Test  and 
Evaluation  (ASTE),  Multiple  linear  and  logarithmic  regres- 
sion are  probably  major  analytical  tools.   Since  data  are 
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not  readily  available  from  a  single  source  to  the  detail 
required  for  econometric  studies,  it  is  necessary  to 
establish  an  operational  data  structure  that  can  be  used  as 
a  basis  for  this  research  and  facilitate  its  data  collec- 
tion. 

This  research  also  applies  the  concepts  defined  in 
system  analysis  and  design  to  implement  decision  support 
software  for  ASTE.  Such  an  implementation  would  greatly 
help  cost  analysts  interact  with  the  complex  and  diverse 
data  base.  Consequently,  the  reliability  cf  their  cost 
estimation  would  be  enhanced,  or,  at  least,  limitations  of 
estimating  methods  clearly  understood. 

Interviews  with  defense  contractors,  Patuxent  River 
Naval  Air  Test  Center,  Edwards  Air  Force  Base  Flight  Test 
Center,  and  on-site  data  collection  are  envisaged  for  the 
first  part  of  the  research.  Utilization  of  statistical 
software  packages  to  identify  elasticities  of  cost  drivers 
will  constitute  the  second  part  of  the  research. 

B.   RESEARCH  METHODOLOGY 

As  stated  in  the  previous  section,  this  research 
encompasses  both  analytical  and  empirical  studies.  The 
methodology  adopted  for  this  research  follows  a  three-step 
process  shown  in  Figure  4-1.  The  activities  of  each  of  the 
three  steps  are  briefly  described  below. 


52 


1.  Exploration  Phase:  This  phase  includes  the  initial 
statement  of  the  problem,  the  assessment  of  its 
importance  and  feasibility,  and  the  establishment  of 
contacts  with  access  to  relevant  data. 

2.  Analysis  Phase:  This  is  an  analytical  approach  to 
problem  solving  that  requires  the  clarification  of 
goals,  definition  of  objectives  and  identification  of 
systematic  research  activities  to  achieve  these  goals. 

3.  Refinement:  This  constitutes  an  iterative  process  to 
gradually  readjust  actions  required  to  better  meet 
stated  objectives.  Particularly,  this  process  would 
highlight  the  sensitivity  of  decisions,  the  values  of 
the  key  cost  drivers  and  assumptions  on  which  es- 
timates are  base  including  economic,  technical, 
operational,  schedule  and  other  problematic  con- 
siderations. 


I 

Exploj 

ration 

Anal 

ysis 

Refin 

ement 

Figure  4-1.  Phases  of  Research  Methodology 


C.   EXPLORATION 

The  Exploration  Phase  was  generated  by  an  attempt  to 
define   the  scope  and   objectives  centering  on  the  issues  of 
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controlling  and  estimating  test  and  evaluation  costs. 
However,  this  research  team's  focus  was  not  clarified  until 
initial  meetings  with  the  Cost  Analysis  Division  of  NAVAIR, 
held  on  August  7-8,  1986.  Although  there  is  a  need  to 
develop  cost  estimators  for  all  of  the  Systems  Test  and 
Evaluation  elements,  it  was  confirmed  at  that  time  that 
initial  efforts  should  be  confined  to  the  Contractor  Flight 
Test  element,  and  if  feasible,  expand  the  research  to 
include  the  Wind  Tunnel  Article  and  Test,  Static  Article  and 
Test,  and  Fatigue  Article  and  Test  elements.  On-site  visits 
at  the  Naval  Air  Test  Center,  as  well  as  interaction  with  a 
selected  group  of  representatives,  demonstrated  some 
feasibility  of  data  collection  and  identified  potential  cost 
drivers,  particularly  in  the  area  of  aircraft  weight  and 
speed,  and  flight  test  hours.  The  initial  Problem  Defini- 
tion in  Figure  4-2  was  developed  following  these  meetings. 

D.   ANALYSIS 

Based  on  the  initial  problem  definition  (Figure  4-2), 
a  study  was  conducted  to  investigate  the  economic,  political 
and  technical  feasibility  of  attaining  the  defined  objec- 
tives. While  no  insurmountable  problems  in  the  financial 
and  technical  aspects  were  envisaged,  the  accessibility  of 
contractors'  proprietary  data  was  identified  as  a  potential- 
ly insurmountable  political  consideration  that  could 
severely  constrain  the  realization  of  the  intended  research. 
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PROBLEM  DEFINITION 

STATEMENT  OP  SCOPE  AND  OBJECTIVES:  August  7-8,  1986 

BETWEEN:      Naval  Postgraduate  School 
AND:      Naval  Air  Systems  Command 

PROJECT:      Cost  Analysis  for  Aircraft  System  Test  and 
Evaluation  -  Data  Collection. 


PROBLEM: 


OBJECTIVES; 


There  is  an  increasing  requirement  from  high 
levels  that  the  Navy's  aircraft  cost  es- 
timators/analysts provide  explicit  estimates 
for  the  sub-elements  of  aircraft  systems  test 
and  evaluation  efforts.  These  data  are  not 
presently  available  at  the  levels  required  by 
the  Naval  Air  Systems  Command  (NAVAIR). 

Collection  of  data  to  facilitate  NAVAIR ' s 
development  of  cost  estimating  relationships 
for  the  following  test  and  evaluation  elements: 
Contractor  Flight  Test,  Wind  Tunnel  Article  & 
Testr  Static  Article  &  Test  and  Fatigue  Article 
&  Test. 


SCOPE: 


Project  to  be  completed  within  six 
calendar  months  at  a  cost  of  no  more  than 
$45,000.  Availability  and  willingness  of 
contractors  to  provide  data  that  could  be 
considered  proprietary.  Inflexibility  of 
NPS  Officers'  schedules  due  to  curriculum 
requirements . 


PRELIMINARY 


SOLUTION: 

Visits  to  Defense  Contractors'  facilities 
to  discuss  the  problem  and  to  ascertain 
the  accessability  and  availability  of  the 
required  data.  Establish  a  database  to 
facilitate  the  analysis  of  the  data. 


FEASIBILITY  STUDY: 


A  feasibility  study  should  be  conducted 
with  the  results  submitted  within  two 
calendar  weeks.  The  cost  of  the  feasibil- 
ity study  is  included  in  the  project  scope 
and  will  not  exceed  $2000. 


Figure  4-2.   Initial  Problem  Definition 
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To  minimize  the  impact  of  this  political  constraint,  a 
strategy  was  formulated  to  ensure  the  viability  of  the  data 
collection  process  by: 

1.  Identifying  contacts  and  soliciting  their  cooperation 

2.  Conducting  on-site  interviews. 

1.   Identifying  Contacts  and  Soliciting  Cooperation 

NAVAIR  provided  a  list  of  twenty-one  contacts 
throughout  the  defense  industry.  Through  phone  conversa- 
tions and  correspondence  (Appendix  A) ,  this  list  was  refined 
and  expanded  to  those  that  indicated  not  only  an  interest  in 
the  results  of  the  research,  but  a  cooperative  attitude 
concerning  the  request  for  company  data  (Appendix  B)  dealing 
with  System  Test  and  Evaluation. 

In  general,  respondents  were  highly  interested  in 
the  nature  of  the  project  and  indicated  their  willingness  to 
assist  in  data  collection  and  critical  evaluation  of  the 
present  cost  analysis  process.  Those  who  declined  to 
participate  cited  the  following  reasons; 

1.  The  lack  of  available  data 

2.  The  current  political  climate  within  the  industry,  and 

3.  On-going  contract  negotiations. 

The  final  list  consisted  of  thirteen  companies,  which  made 
available  personnel  from  a  wide  range  of  aircraft-related 
disciplines  as  depicted  in  Figure  4-3. 
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Job  Description 

Number  of  Contacts 

Cost  Estimation 
Data  Managers 
Engineers 

18 
7 

11 

Figure  4-3,   Interviewees  Job  Description  Breakdown 


2 .   On-site  Interviews 

Interviews  were  conducted  at  thirteen  locations 
throughout  the  United  States.  Interview  teams  consisted  of 
at  least  two  research  members  per  visit.  Surveying  time 
lasted,  on  the  average,  approximately  three  hours  for  each 
location,  varying  from  forty-five  minutes  to  ten  hours. 

To  prepare  for  the  interviews,  the  research  team 
brainstormed  to  establish  the  following  initial  list  of 
issues  that  appeared  to  be  important  for  the  research 
methodology. 

1.  What  are  the  cost  drivers  in  ASTE? 

2.  How  do  the  characteristics  and  complexity  of  the 
aircraft  relate  to  the  average  engineering  hour? 

3.  What  statistical  and  econometric  models  can  be 
appropriately  applied  to  ASTE?  Is  multiple  regression 
appropriate  to  this  problem? 
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4.  Are  the  military  work-breakdown  structures  pertinent 
to  ASTE? 

5.  How  can  a  computer  system  (hardware  and  software)  be 
designed  and  implemented  to  conduct  testing  of  the 
models? 

6.  How  have  data  been  defined  and  collected  in  different 
test  sites  (military  and  civilian)?  Can  they  be 
filtered,  and  standardized?  What  extrapolation  method 
should  be  used  for  missing  data? 

7.  How  do  these  cost  drivers  affect  the  average  engineer- 
ing hour  per  hour  of  flight? 

8.  Is  there  any  correlation  between  the  number  of 
instrumentation  channels  and  the  number  of  engineering 

hours? 

9.  Is  there  a  linear  cost  relationship  between  different 
types  of  aircraft  with  respect  to  their  complexity  and 
characteristics? 

10.  How  sensible  and  stable  are  the  elasticities  of  the 
cost  drivers? 

11.  What  are  the  implications  of  the  findings? 

12.  Where  is  it  most  cost-effective  to  conduct  future 
military  ASTE — at  the  contractors  facilities  or  at  the 
established  flight  test  centers.  Is  the  answer  the 
same  for  both  flight  test  centers? 

13.  Do  the  findings  have  an  impact  on  the  service's  opera- 
tional test  and  evaluation? 

14.  How  does  inflation  affect  the  predictive  power  of  the 
findings? 

This  initial  list  of  questions  was  sent  to  interested 

parties.   Additionally,  to  improve  our  own  understanding  of 

industry  practices  and  procedures,  the  list  of  issues  were 

distilled  to  the  following  set  of  questions  that  could  be 

addressed  in  on-site  interviews. 

1.  Would  you  provide  general  specifications  of  all  your 
company's  aircraft  to  include:  aircraft  type  and  mis- 
sion, thrust  to  weight  ratio,  speed,  size,  ceiling, 
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combat  radius,  commonality/GFE  between  models,  and  use 
of  exotics  and/or  composites? 

2.  What  methodologies  do  you  use  for  accounting  and 
record  keeping  of  cost  items  (development  and  test- 
ing)? 

3.  How  are  these  accounting  and  record  keeping  items 
defined?  Are  these  definitions  standardized  company 
wide?  Do  you  have  a  computerized  cost  accounting 
system? 

4.  What  methodologies  do  you  use  to  extrapolate  cost 
estimators  from  these  data? 

5.  In  the  area  of  avionics,  what  impact  do  variations  in 
the  following  items  have  on  the  labor  hours  and  flight 
test  hours  of  your  aircraft  systems: 

a.  The  number  of  instrumentation  channels. 

b.  Number  of  lines  of  delivered  software  code. 

c.  The  number  of  separate  CPU's  installed  in  the 
aircraft. 

d.  New  electronic  technology  (e.g.,  VSLI ) . 

e.  What  hardware  integration  factors  affect  the 
cost  (i.e.,  is  a  decentralized  system  more  cost- 
effective  than  a  centralized  system?). 

f.  Power   usage. 

6.  What  do  you  believe  are  the  most  important  cost- 
drivers?    Could   you   prioritize    them? 

7.  If  due  to  cost  constraints  you  had  to  reduce  test  and 
evaluation  activities,  what  would  be  your  primary 
candidates  for  elimination?  Which  activities  do  you 
consider   indispensible   to   test   and   evaluation? 

8.  What   would   you   do   to   reduce   costs? 

This  set  of  questions  permitted  a  cross  comparison  among 
companies.  More  interesting,  it  was  found  that,  as  the 
interview  progressed,  the  domain  knowledge  of  the  research 
team    expanded    exponentially    which    improved    the    quality    of 
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the  remaining  discussions.  The  contacts  below  also  raised 
numerous  issues  they  thought  significant  enough  to  included 
in  the  research. 


12  November  1986 

12  November  198  6 

13  November  1986 

13  November  1986 

14  November  1986 
17  November  1986 
17  November  1986 

17  November  1986 

18  November  1986 
18  November  1896 

18  November  1986 

19  November  1986 

20  November  1986 


Rockwell  International  Corporation 
Los  Angeles,  California 

Boeing 

Seattle,  Washington 

Air  Force  Flight  Test  Center 
Edwards  Air  Force  Base,  California 

McDonnell  Douglas  Corporation 
Long  Beach,  California 

Lockheed  California  Company 
Burbank,  California 

Rockwell  International 
Columbus,  Ohio 

General  Dynamics  Corporation 
Fort  Worth,  Texas 

LTV  Aerospace  and  Defense 
Dallas,  Texas 

Grummann  Aerospace  Corporation 
Bethpage,  New  York 

Fairchild  Aircraft 
Farmingdale,  New  York 

McDonnell  Aircraft  Company 
St.  Louis,  Missouri 

Lockheed  Georgia  Company 
Marietta,  Georgia 

Naval  Air  Test  Flight  Center 
Patuxent  River,  Maryland 


As  the  interview  process  continued,  important  issues 
converged  demonstrating  a  industry-wide  commonality  of 
problems  (see  Chapter  V). 
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E.   REFINEMENTS 

A  post-interview  meeting  was  convened  at  NAVAIR  to 
assess  the  results  of  the  contractor  visits.  The  research 
team  reported  to  the  sponsor  the  nature  and  amount  of  data 
that  were,  and  could  be,  collected.  Also,  the  sponsor  was 
briefed  on  the  areas  considered  most  important  by  contrac- 
tors (  see  Chapter  V),  which  they  had  recommended  be  added 
to  the  present  cost  process  to  make  it  more  reliable  and 
accurate,  in  particular: 

1.  Software 

2.  Avionics 

3.  Instrumentation  channels 

4.  Requirements  to  meet  performance  guaranties 

5.  Contract  warranties  required  by  the  Government 

6.  Fixed-priced  contracts  and 

7.  Combined  DOD  /  Contractor  Flight  Testing. 

It  became  obvious  that  these  new  dimensions  could 
contribute  to  the  rapid  growth  of  the  present  project  scope. 
If  they  were  all  included,  it  would  no  longer  be  feasible  to 
accomplish  the  totality  of  the  project  given  the  constraints 
imposed.  Therefore,  the  team  requested  the  sponsor  to 
refocus  the  problem  and  set  research  priorities  in  light  of 
these  new  factors.  It  was  then  stated  that  the  focus  would 
remain  on  Contractor  Flight  Test,  with  Wind  Tunnel  Test, 
Static  Test,  and  Fatigue  Test  studies  as  secondary  goals. 
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In  order  to  achieve  this  restated  goal,  interpreta- 
tion, review,  and  standardization  of  data  became  indispen- 
sible.  Due  to  his  expertise,  the  sponsor  offered  his 
assistance.  This  operation  is  time  intensive,  requiring 
tedious  and  systematic  efforts.  In  the  meantime,  a  follow- 
up  letter  was  sent  to  contractors  seeking  time-phased  data 
(Appendix  C).  In  parallel,  a  data  base  was  designed  to 
accommodate  the  data  received  from  the  contractors  (see 
Volume  II,  Chapter  III). 


62 


V.   COST  DRIVERS  IN  SYSTEM  TEST  AND  EVALUATION 

A.  INTRODUCTION 

Past  studies  have  primarily  considered,  weight,  speed 
and  the  number  of  aircraft  as  the  most  statistically 
significant  cost  estimators.  In  fact,  other  cost  drivers — 
such  as  avionics,  software,  management  strategies--have 
recently  emerged  as  important,  if  not  more  important  than 
these  three  drivers.  Unfortunately,  due  to  the  current 
practices  of  data  collection,  it  is  impossible  to  statically 
quantify  the  new  cost  drivers.  The  only  approach  that 
remains  appropriate  to  evaluate  the  importance  of  these 
drivers  is  to  use  the  Delphi  technique  to  solicit  expert 
opinions  in  the  aircraft  system  cost  estimation  area. 
Interviews  with  aerospace  engineers,  pilots.  Test  and 
Evaluation  managers  from  Defense  Contractors  and  DOD,  test 
flight  centers,  and  defense  contractors  have  been  conducted 
to  brainstorm  issues  regarding  Aircraft  Systems  Test  and 
Evaluation  costs. 

B.  IDENTIFICATION  OF  THE  MOST  IMPORTANT  COST  DRIVERS 
1 .   Mission  as  the  Determinant  of  Cost  Drivers 

Mission  is  the  cornerstone  upon  which  any  major 
program  is  built.    Until  the  mission  of  the  proposed 
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aircraft  system  is  defined,  it  is  not  possible  to  determine 
specifications;  even  at  a  broad  level.  However,  once  the 
mission  is  known,  then  the  many  factors  that  form  the 
complex  world  of  Defense  Systems  Development  and  Acquisition 
begin  to  come  together.  Many  of  these  factors  are  clearly 
defined  and  are  measurable  cost  drivers  that  will  be 
discussed  later  in  this  chapter.  It  is  important  to  note, 
however,  that  there  exists  an  environment  within  which  DOD 
and  Defense  Contractors  operate  that  effects  all  programs  as 
surely  as  the  more  obvious  cost  drivers. 

One  of  the  major  managerial  concerns  is  whether  a 
bid  can  be  submitted  in  such  a  way  that  it  will  not  only 
allow  the  company  to  develop  and  produce  the  system  within 
the  defined  financial  constraints,  but  insure  that  there  is 
a  means  available  to  recoup  the  cost  of  DOD's  "inevitable" 
modifications  to  the  specifications  and/or  performance 
guarantees.  It  is  no  longer  a  matter  of  "if"  there  will  be 
changes,  but  rather  a  question  of  "when  and  how  many".  The 
Cost  Plus  Contract  provided  the  contractor  with  an  insurance 
clause,  which  often  proved  to  be  very  expensive  from  the 
government's  perspective.  The  Fixed  Price  Contract  now 
mandated  by  DOD  has  removed  this  insurance  clause,  provided 
DOD  with  protection  from  cost  overruns,  and  has  moved  the 
"risk  factor"  to  the  contractor.  Since  it  should  not  be 
considered  unreasonable  for  either  DOD  or  the  contractor  to 
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expect  some  changes  over  the  long  developmental  life  cycle 
of  a  major  weapon  system,  the  present  relationship  is  one 
punctuated  with  compromise  and  conciliation.  If  it  were 
otherwise,  the  development  and  attainment  of  a  program's 
performance  objectives  would  be  impossible. 

The  requirement  to  provide  warranties  on  an  aircraft 
system  that  has  yet  to  be  built,  and  whose  operational  life 
is  unknown,  is  another  major  management  concern  in  the  early 
stages  of  a  program.  How  does  an  analyst  quantify  costs, 
for  inclusion  in  a  bid,  that  are  so  unpredictable?  And  how 
does  management  calibrate  the  experience  and  expertise  of 
its  engineers  in  cost  estimators?  Are  the  cost  estimates  as 
accurate  as  the  specifications  and  performance  guarantees? 

Sub-contractors  and  secondary  vendors  are  often 
trapped  into  making  guarantees  too  early  in  the  process. 
Their  desire  to  be  a  part  of  the  program  leaves  them  with  no 
other  alternative,  and  as  a  result  the  integrated  overall 
mission  is  typically  not  met. 

When  a  ceiling  dollar  value  is  included  in  a  Request 
for  Proposals  there  are  few  contractors  who  will  refrain 
from  bidding  the  indicated  ceiling,  even  if  their  own  cost 
estimators  believe  that  it  will  be  impossible  to  stay  below 
that  figure.  "If  we  get  our  foot  in  the  door,  we'll 
renegotiate  later." 
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2.   Aircraft  Weight 

All  companies  interviewed  indicated  that  weight 
still  constitutes  a  major  cost  driver  in  system  development. 
Because  of  the  availability  of  the  data  on  weights,  it  has 
been  relatively  easy  to  use  weight  as  a  cost  estimator. 
Dollars  and  hours  per  pound  have  often  been  used  as  es- 
timators. Everything  else  being  equal,  the  larger  the 
aircraft  the  greater  the  cost,  i.e.,  a  bomber,  which  is 
approximately  five  times  heavier  than  a  fighter,  has  overall 
costs  (Level  2  of  the  WBS)  that  are  roughly  five  times  more 
than  that  of  the  fighter.  This  interpretation  corresponds 
to  earlier  studies  performed  by  PRC  (1967),  RAND  (1972, 
1975,  1976),  and  Noah  (1973).  These  studies  all  ranked 
weight  as  the  primary  cost  driver.  As  weight  increases, 
cost  increases  too.  However,  as  weight  increases,  at  some 
point,  cost  per  pound  decreases.  Additionally,  aircraft 
density  effects  the  cost  per  pound  of  aircraft.  As  an 
aircraft  is  loaded  with  more  sub-systems  (e.g..  Avionics), 
cost  per  pound  of  the  aircraft  increases. 

Recent  use  of  composites  has  altered  the  statistical 
quality  of  weight  as  a  cost  estimator  since  historical  data 
are  associated  with  metal.  It  has  been  hypothesized  that  as 
more  experience  is  gained  with  composites,  weight  will  again 
become  a  more  stable  and  predictable  factor  because  of  the 
nature  of  building  with  composites.  Although  testing  an 
aircraft  component  built  entirely  with  composites  should  be 

66 


easier,  one  of  the  major  uncertainties  when  dealing  with 
composites  in  the  Contractor  Flight  Test  phase  occurs  when 
an  test-site  modification  or  repair  to  an  aircraft  is 
required.  With  the  increased  use  of  composites  in  the 
future,  weight  is  expected  to  move  from  labor-intensive  to  a 
automation(capital)-intensive  factor.  While  it  is  still 
difficult  to  predict  at  the  present  time  how  the  use  of 
composites  will  drive  the  cost  associated  with  weight,  it  is 
plausible  that  developmental  testing  will  progressively 
become  less  expensive.  It  is  also  hard  to  relate  weight  of 
composites  to  cost  since  there  are  more  parts  in  non- 
composite  aircraft  than  in  composite  aircraft. 
3.   Aircraft  Speed 

After  weight,  speed  is  also  recognized  as  an 
important  cost  drivers.  However,  it  was  identified  only  by 
70%  of  the  opinions  solicited.  Thrust  being  constant, 
weight  is  related  to  speed.  One  of  the  reasons  that  speed 
represents  an  important  cost  driver  is  the  fact  that  due  to 
the  requirement  to  meet  performance  guaranties,  development 
costs  increase  at  a  spiraling  rate  as  engineers  work  with 
the  aircraft  at  critical  speeds.  The  reduced  total  weight 
of  an  aircraft  due  to  the  use  of  composites  will  decrease 
the  thrust  required  to  attain  the  same  speed  of  a  non- 
composite  aircraft,  and  as  a  consequence,  developmental  cost 
in  this  area  should  be  diminished. 
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4.   Avionics  Complexity 

With  one  exception,  avionics  complexity  was  another 
cost  driver  that  emerged  from  the  interviews.  Advanced 
electronic  technology  has  resulted  in  improved  reliability 
of  avionics  and  substantial  amelioration  in  troubleshooting 
and  maintenance  procedures.  Test  effectiveness,  test 
vehicle  flight  frequency,  and  data  validity  have  greatly 
improved.  Many  contractors  claimed  that  achievement  in 
avionics  have  been  responsible  for  their  recent  gain  in 
program  planning  effectiveness  and  sortie  rates  of  their 
flight  test  programs. 

While  software-intensive  avionics  substantially 
innovates  the  capabilities  of  aircraft  systems,  it  also 
presents  enormous  testing  requirements.  For  example,  a  high 
percentage  of  test  work  accomplished  by  the  Air  Force  for 
the  F-15,  F-16,  and  B-l  is  related  to  software.  When  the 
volume  allocated  to  avionics  within  the  airframe  is  con- 
stant, advanced  technology  decreases  the  weight  and  in- 
creases the  avionics  capabilities.  This  implies  that  while 
more  complex  aircraft  can  be  built  without  affecting  the 
costs  related  to  increased  weight  and  speed,  the  complexity 
of  avionics  will  drive  the  total  system  costs  up.  Avionics 
is  a  difficult  cost  to  estimate  due  to  the  difficulty  in 
estimating  software. 
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5 .   Software 

Indeed,  an  important  factor  that  has  emerged  from 
the  increased  avionics  complexity  is  software.  Unlike 
weight,  there  is  a  predominant  concern  on  developing  the 
ability  to  accurately  estimate  software  costs.  This  element 
is  regarded  as  the  most  difficult  cost  driver  to  estimate. 
First,  there  is  little  consistency  among  companies  in  the 
way  aircraft  system  software  is  developed.  Some  develop  by 
total  system  concepts,  others  modularize  by  functional  areas 
(e.g.,  navigation,  fire  controls,  weapons),  and  still  others 
concentrate  on  implementing  interfaces  between  blackboxes 
developed  by  various  sub-contractors.  Second,  estimation 
methods  range  from  dollars-per-word  to  parametric  models 
based  on  total  lines  of  codes.  These  models  include  COCOMO, 
SLIM,  RCA/PRICE/S,  IBM  Walston-Felix ,  and  Boeing  Computer 
Service.  (Wolverton,  1980)  However,  the  complete  lack  of 
historical  data  and  experience  with  Aircraft  System  Test  and 
Evaluation  software  makes  it  virtually  impossible  to  apply 
these  parametric  models.  (For  a  survey  of  parametric  models, 
see  Volume  II,  Chapter  II.)  The  estimation  problem  is 
further  complicated  by  the  fact  that  software  costs  depend 
on  the  number  of  modules,  the  size  of  the  modules,  the 
degree  of  integration  complexity,  and  the  number  of  proces- 
sors in  distributed  systems  available  to  run  the  integrated 
software  system. 
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It  is  felt  that  the  mission  determines  the  software 
costs.  For  example,  a  training  mission  does  not  require  a 
complex  software  structure  as  compared  to  an  all-weather 
attack  mission.  Consequently,  software  costs  for  the 
trainer  are  much  less  expensive  than  for  those  of  the  all- 
weather  attack  aircraft.  In  addition  to  the  primary 
mission,  the  specific  number  of  mission  tasks  the  aircraft 
is  capable  of  performing  does  play  an  important  factor.  For 
example,  in  general,  the  ECM  mission  would  derive  a  higher 
software  cost  than  the  ASW  mission.  However,  the  software 
cost  of  the  AWACS  is  higher  than  the  E-2C.  These  two 
aircraft  perform  similar  missions  but  on  different  scale. 
Another  comparison  would  be  between  the  F/A-18  and  the  A-6. 
If  both  aircraft  were  built  from  the  ground  up  using  today's 
technology,  the  software  cost  of  the  multi-mission  F/A-18 
would  probably  be  substantially  higher  than  the  A-6. 

A  strategy  to  deal  the  increasing  cost  of  testing 
software  intensive  system  is  to  use  simulation.  Research 
conducted  at  the  Air  Force  Flight  Test  Center,  Edwards  Air 
Force  Base  argues  that: 

1.  Three-fourths  of  all  software  problems  are  resolvable 
on  the  ground  at  a  fraction  of  a  cost  of  a  flight 

2.  A  reduction  in  test  flying  hours  translates  into  a 
reduction  of  test  costs  and  an  acceleration  of  the 
test  schedule 

3.  The  costly  and  inefficient  fly-fix-fly  approach  is 

minimized 
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4,  Ground  testing  is  more  efficient  than  flight  testing 
because  the  experiment  is  controlled,  repetition  of 
test  conditions  is  rapid  and  simple 

5.  Flight  test  time  is  used  more  effectively  by  isolat- 
ing/keying on  risks  areas  and  smarter  profile  plan- 
ning. (Adolf  and  Montgomery,  1985,  p. 3) 

Tables  5-1  and  5-2  show  expected  cost  savings  for  a  typical 
fighter  test  program  utilizing  simulation  of  software- 
intensive  systems. 

A  second  approach  to  resolving  the  problem  of 
software  cost  estimation  would  be  for  the  entire  Defense 
Industry  to  officially  endorse  the  utilization  of  structured 
analysis  and  design  methodology  through  functional  decom- 
position. (Yourdon,  1986)  Such  decomposition  by  functions 
would  make  it  easier  for  software  developers  to  capture  the 
complexity  of  each  function  and  systematically  integrate 
them  as  a  total  system  for  cost  estimation.  Similar 
software  engineering  techniques  and  experience  will  result 
in  more  standardized  data  collection  needed  for  future 
estimation. 

Another  potential  area  of  improvement  is  the 
implementation  of  distributed  systems  using  parallel 
processors.  While  no  statistical  evidence  was  gathered  to 
assess  the  impact  of  separate  CPUs  on  aircraft  testing,  the 
majority  of  interviewees  believe  that  the  conduct  of  tests 
and  the  probability  of  completion  during  a  specific  flight 
were  enhanced  when  several  CPUs  were  installed,  permitting 
the   successful  testing  of  some  aircraft  systems  when  others 
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TABLE  5-1 


AIRCRAFT 

TIST   FLIGHT 
FREQUENCY 
(FLIGHTS /MONTH  J 

TIST   AIRCRAFT 

OPERATIONS   COST 

(51000/HOUR) 

?UCHT  TEST   FIXED 
SUPPORT   COST 
(S 1000 /MONTH) 

PROCRA^OtASLE 

CAPACITY 
(1000    WORDS) 

CURRENT 

GENERATION 

FIGHTER 

10 

20 

SI, 000 

300    -    700 

CURRENT 

GENERATION 

aOMBER 

5 

100 

52,000 

600    -    300 

NEXT 

GENERATION 
FIGHTER 

10 

40 

31.500 

1. 000 

TABLE  5-2 


COST  OF  FLIGHTS: 

(250  FLIGHTS)  (1.25  HRS/FLT)  ($20.000/HR) 
LESS  COST  OF  SIMULATION: 

(250  FLIGHTS)  (1  HR/FLT)  ($4,000/HR) 
NET  SAVINGS  FROM  REDUCED  FLIGHTS: 
(20  AIRCRAFT  MONTHS)  ( SI ,000  ,000/MONTH) 

TOTAL  SAVINGS:   55,250,000  ♦  520,000,000 


5  6,250.000 


5  5,250.000 
$20,000,000 

525,250.000 
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have  failed.   In  the  past,  a  central  CPU  failure  precluded 
continuation  of  the  testing  process. 

6.  Power  Supplies 

Power  supplies  play  a  vital  role  in  the  operation  of 
all  aircraft.  These  power  sources  provide  a  range  of 
outputs  depending  on  the  requirements  and  number  of  the 
devices  they  support.  This  accounts  for  the  varied  capabil- 
ities, sizes  and  quantities  of  power  supplies  that  can  be 
found  in  any  aircraft  system. 

The  majority  of  the  volume  and  weight  of  a  typical 
power  unit  is  taken  up  by  the  filters  required  to  ensure 
that  delivered  current  is  within  specified  operating  ranges. 
When  compared  to  the  technological  advances  made  in  other 
avionics  areas,  it  is  obvious  that  there  has  been  very 
little  progress  in  the  area  of  reducing  or  eliminating  these 
unit  filters,  thereby  improving  the  overall  efficiency, 
presently  necessary  to  meet  the  requirement  specifications. 
If  more  efficient  power  units  were  developed,  the  number  of 
these  sources  could  be  reduced,  thereby  reducing  the  total 
weight  and  volume  of  avionics  in  the  aircraft.  Also,  if 
power  sources  were  more  stable  and  cleaner,  testing  costs 
would  be  reduced. 

7 .  Data  Reduction 

Data  reduction,  which  is  the  process  of  reviewing 
and  analyzing  the  data  gathered  during  testing,  presents 
another    area    of    concern.       Compounding    this    problem    is    the 
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increased  number  of  instrumentation  channels.  With  the 
advent  of  a  central  bus  within  the  airframe,  engineers  have 
recognized  the  potential  for  monitoring  additional  data 
points  that  were  not  previously  feasible.  Convinced  that 
these  data  could  now  be  gathered  relatively  easily,  they 
requested  that  procedures  for  monitoring  additional  data 
points  be  included  in  the  test  plan.  Most  contractors  felt 
that  this  process  would  be  cost  effective  since  it  would 
provide  a  wide  cross  section  of  data  on  each  and  every 
flight  which  would  allow  maximum  concurrent  testing  regard- 
less of  the  primary  purpose  of  that  flight. 

However,  the  current  technology  that  assists  in 
analyzing  data  has  not  progressed  as  fast  as  technology's 
ability  to  produce  raw  data.  Collected  data  have  become  so 
voluminous  and  overwhelming  to  the  extent  that  it  has  become 
impossible  for  the  engineers  to  properly  analyze  all  of  the 
data  unless  the  time  constraints  of  the  test  program  were 
extended. 

8.  The  Number  of  Test  Aircraft 

The  number  of  test  aircraft  also  constitutes  an 
important  cost  element.  Historically,  it  was  observed  that 
this  cost  driver  was  considered  one  of  the  most  significant 
in  flight  test  cost  estimation,  along  with  weight  and  speed. 
(PRC,  1967;  RAND,  1972;  RAND  1975)  As  the  number  of 
aircraft  increases,  the  number  of  separate  test  flight 
augments  and  drives  the  total  cost  higher.   aircraft  are 
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instrumented  to  perforin  specific  portions  of  the  test  plan 
and  therefore  cross  utilization  is  infeasible  unless 
aircraft  are  modified  for  other  specific  tests.  The 
determination  of  an  optimal  number  of  aircraft  is  extremely 
difficult  since  the  cost  relationship  is  not  linear.  If  the 
number  of  aircraft  is  below  the  testing  capacity  then 
testing  resources  are  wasted.  Conversely,  when  the  number 
of  aircraft  exceeds  the  testing  capacity,  there  is  a  loss  of 
the  maximum  utilization  of  test  aircraft  assets  and  ex- 
perienced personnel.  As  an  attempt  to  reduce  costs, 
contractors  argue  that  the  number  of  aircraft  used  in  flight 
testing  should  be  held  to  the  minimum  required  to  meet  the 
test  plan  schedule.  They  felt  that  DOD  as  a  whole  should  not 
have  as  much  input  in  determining  the  number  of  test 
aircraft  as  it  is  presently  exercised. 
9 .   Delivery  Schedule 

It  is  expected  that  this  cost  element  is  gaining  in 
importance  due  to  the: 

1.  Increasing  complexity  of  aircraft 

2,  Requirements  to  meet  all  performance  guaranties  imposed 
by  the  Government  prior  to  the  first  delivery  of  pro- 
duction aircraft. 

As  a  proposition  to  reduce  this  problem,  it  was  proposed  by 

several  contractors  that  aircraft  be  delivered  in  a  block 

(phase)  program.   A  block  program  would  consist  of  quickly 

delivering  a  small  number  of  operational  aircraft  capable  of 

meeting  at  least  80%  of  mission  requirements.   As  the 
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aircraft  are  used  by  operational  units,  additional  feedback 
would  be  inputted  to  improve  the  performance  of  subsequent 
production  blocks  of  aircraft.  Operational  input  to  the 
test  program  would  insure  that  the  aircraft,  as  designed, 
are  in  fact  meeting  real  needs.  This  would  allow  contrac- 
tors to  continue  utilizing  test  aircraft  to  solve  any 
problems  which  inhibited  the  attainment  of  all  performance 
guaranties,  without  delaying  the  delivery  of  operational 
aircraft  to  DOD.  Although  this  would  most  certainly  result 
in  a  requirement  to  retrofit/modify  the  initial  block  of 
aircraft,  it  is  felt  that  this  procedure  would  ultimately 
provide  a  more  capable  aircraft  to  operational  units  in  a 
more  timely  manner.  However,  this  strategy  would  create  a 
potential  problem  in  the  upgrading  of  the  early  delivered 
aircraft.  Usually,  the  cost  of  modification  is  significant- 
ly greater  than  the  cost  of  initial  production. 
10 .   Joint  Contractor/Military  Testing 

Various  degrees  of  joint  testing  have  been  prac- 
ticed by  DOD.  It  consists  of  combining  the  efforts  of  the 
contractors  and  the  military  in  the  parallel  testing.  These 
combined  efforts  include  the  development,  test,  and  evalua- 
tion phases.   Advantages  of  this  approach  include: 

1.  Early  involvement  of  operational  aircrew  and  sharing 
of  test  data  from  the  genesis  of  the  test  program 

2.  Reduced  duplication  of  efforts 

3.  A  new  critical  perspective  of  the  progress  of  the 
program  that  could  help  identify  shortcomings  or 
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problem  areas  in  the  early  stages  of  the  test  program 
that  otherwise  might  not  have  been  identified  until 
the  Operational  Test  and  Evaluation  phase;  this  would 
lead  to  substantial  savings  in  time  and  costs. 

Contractors  feel  that  they  no  longer  have  primary 
control  over  their  own  flight  test  programs.  The  require- 
ment to  use  government  test  sites  and  facilities,  which  are 
inadequately  equipped  to  support  simultaneous  test  programs 
of  major  aircraft  systems,  is  viewed  as  counterproductive. 
These  inadequacies  require  that  program  priorities  be 
established  which  results  in  test  delays  and  some,  if  not 
all,  of  the  following  ramifications  to  the  contractors: 

1.  They  have  to  relocate  their  personnel  and  test 
equipment  to  governmental  facilities 

2.  They  have  to  train  military  test  pilots  in  highly 
instrumented  test  aircraft 

3.  Flight  schedules  must  be  submitted  to  the  military  for 
approval  before  actual  real  time  planning  can  begin 

4.  They  are  subject  to  DOD  rules  and  regulations,  which 
many  find  too  restrictive 

5.  Their  schedules  are  often  reviewed  by  military 
personnel  who  may  not  be  familiar  with  the  intricacies 
of  the  contractor's  flight  test  program.  The  explana- 
tions and  briefings  required  to  clear  up  these 
misunderstandings  cause  unnecessary  interruptions  in 
the  test  program 

6.  The  turnover  of  key  military  personnel  during  the  Test 
and  Evaluation  programs  often  precipitates  revisions 
of  previously  reviewed  and  approved  cockpit/aircraft 
configurations . 

From  the  contractors'  viewpoint,  the  requirements 

specifications  initially  established  at  the  beginning  of  the 

program  are  constantly  modified  during  the  System  Test  and 

Evaluation  life  cycle.   Contractors  contend  that,  given 
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accurate,  detailed,  and  complete  DOD  specifications,  they 
could  complete  the  required  Developmental  Test  and  Evalua- 
tion more  effectively  and  efficiently  without  interference 
from  external  agencies.  As  such,  the  contractors  would 
ultimately  be  responsible  for  development  testing,  test 
management,  and  more  important,  test  success. 

It  is  important  to  note  that  opinions  in  this  issue 
have  been  extremely  bimodal.  Historical  data  could  be  found 
to  support  both  sides  of  the  issue  as  both  management 
strategies  have  been  successful  in  producing  high  quality 
aircraft  for  the  Department  of  Defense.  Since  these 
arguments  cover  only  a  partial  view  of  the  entire  problem, 
further  studies  are  necessary  to  determine  which  strategy  is 
capable  of  producing  the  highest  quality  and  most  cost- 
efficient  aircraft.  Specifically,  a  more  integral  approach 
appears  to  be  the  only  way  to  assess  the  total  impact  of  the 
two  strategies. 

11.   Political 

Last  but  not  least.  Congressional  funding  delays 
are  cited  as  having  a  non-trivial  impact  on  the  total  system 
cost.  These  delays  caused  an  unnecessary  time  lag  between 
the  development  and  production  phases.  This  lack  of  im- 
mediate follow-up  results  in  loss  of  experience  and  exper- 
tise gained  in  the  prototyping  phase  and  serious  economical 
and  personnel  problems  in  maintaining  qualified  manpower. 
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C .   SUMMARY 

The  interviews  conducted  with  Defense  Contractors  and 
the  military  test  centers  resulted  in  numerous  recommen- 
dations for  improvement  of  the  present  process  of  System 
Test  and  Evaluation.  While  the  following  summarized  list  of 
recommendations  addressed  in  this  chapter  represents  neither 
an  exhaustive  nor  final  list  of  possible  actions,  they  do 
highlight  several  areas  of  major  concern: 

1.  The  use  of  simulation  in  software  testing 

2.  The  utilization  of  a  structured  analysis  and  design 
methodology  to  develop  software 

3.  The  implementation  of  distributed  systems  using 
parallel  processors 

4.  The  development  of  more  efficient  power  units, 
thereby  reducing  the  total  weight  and  volume  of 
avionics  in  the  aircraft 

5.  The  reduction  of  the  required  number  of  test 
aircraft 

6.  The  delivery  of  aircraft  in  a  block  (phase)  program. 
Also,  further  studies  are  necessary  to  determine  if  joint 
testing  is  producing  the  highest  quality  and  most 
cost-efficient  aircraft. 
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APPENDIX  A:   CONTRACTOR  FORM  LETTER  #1 


Naval  Postgraduate  School 
Monterey,  Ca.   93943 
October  24,  1986 


Dear 


I  have  extracted  some  information  from  our  thesis  proposal 
to  give  you  a  better  idea  of  what  this  project  entails  and  will 
try  to  accomplish.   I  have  also  enclosed  a  tentative  format 
depicting  a  Work  Breakdown  Structure/Functional  Cost  Element 
Matrix  which,  with  some  flexibility,  will  attempt  to  use  in  our 
data  collection  effort.   I  will  call  you  on  the    rd  to  answer 
any  questions  you  may  have  concerning  the  project  and  provide  you 
an  update  of  our  latest  discussions  with  NAVAIR. 

The  week  of  November  16th,  we  will  be  making  a  cross-country 
collection  trip  to  all  of  the  defense  contractor  facilities 
involved  in  this  project.   We  would  like  to  meet  with  you  in 
Columbus,  Monday  morning,  November  17th,  to  collect  the  initial 
data  you  have  extracted,  and  discuss  the  project  in  more  detail. 
We  realize  that  doesn't  give  you  very  much  time,  but  we  are 
trying  to  have  most  of  the  initial  data,  from  all  sources, 
collected  and  reviewed  for  discussion  at  a  NAVAIR  meeting  in 
Washington  D.  C.  November  20th.   This  meeting  will  be  used  to 
finalize  the  focus  of  the  data  needed  to  complete  this  research 
project.   We  will  contact  you  immediately  upon  our  return  from 
the  NAVAIR  meeting  to  advise  you  of  the  results  of  the  meeting 
and  to  eliminate  any  ambiguities  concerning  the  data  we  received 
and  finalize  our  further  data  collection  requirements.   We 
anticipate  two  follow-on  trips.   The  first  for  final  collection 
of  data,  the  second  to  present  the  results  of  our  research  and 
copies  of  our  cost  estimating  tools. 

The  basic  data  we  will  need  will  be  the  direct  labor  hours 
and  flight  hours  related  to  all  DOD  aircraft  systems  your 
facility  helped  produce. 
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The  information  we  require  necessitates  a  breakdown  of 
Contractor  Flight  Test,  Wind  Tunnel  Test,  Static  Test,  and 
Fatigue  Test  into  the  lowest  available  indentured  sub-elements 
that  your  company  used  during  system  test  and  evaluation. 
It  is  important  to  our  efforts  that  we  obtain  the  Contractor 
Flight  Test  labor  hours,  in  particular,  in  a  time-phased  fashion 
along  with  time-phased  flight  hours,  so  as  to  be  able  to  relate 
the  two.   We  would  also  like  to  obtain  time-phased  cost  data  for 
the  other  test  elements  if  possible. 

We  would  like  to  meet  with  the  individuals  who  develop  the 
initial  Work  Breakdown  Structure  (WBS)  estimates  to  discuss  their 
ideas  on  what  factors  became  the  cost-drivers.   If  available, 
inputs  from  flight  test  engineers  would  contribute  significantly 
to  the  balance  of  the  data  we  are  collecting. 

I  realize  that  we  are  asking  for  what  you  may  consider  to 
be  part  of  your  company's  proprietary  information.   I  assure  you, 
in  the  strongest  possible  terms,  that  all  data  will  be  held  in 
the  strictest  confidence,  used  only  for  the  generation  of  a  cost 
estimating  algorithm,  and  not  divulged  outside  of  the  government. 


We  appreciate  your  cooperation  and  look  forward  to  working 
together  in  the  coming  months. 


Semper  Fi , 


William  J.  Foster     David  F.  Lee 
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A.  PROJECT  TEAM 

Dr.  Tung  X.  Bui  Assistant  Professor   NPS 
LCDR  Robert  W.  Fonnesbeck   529-66-1259   USN 
Major  William  J.  Foster   537-50-0275   USMC 
Major  David  J.  Lee   568-74-7785   USMC 
LT  Walter  J.  Moore   273-42-9741   USN 

Com:   408-646-2995/2630 
Av:        878-2995/2630 

B.  DISCUSSION 

Current  budget  constraints  have  resulted  in  an  increasing 
emphasis  on  the  cost  effective  procurement  of  aircraft 
systems  within  the  Department  of  Defense.   Of  late,  the  Cost 
Analysis   Division  of  the  Naval  Air  Systems  Command 
(NAVAIR)  has  received  numerous  requests  to  provide  explicit 
estimates  for  the  sub-elements  of  aircraft  system  test  and 
evaluation  efforts.   These  data  represent  lower  levels  in 
the  Work  Breakdown  Structure  than  previously  required.   This 
additional  data  and  the  methods  necessary  to  produce  them 
are  needed  in  order  to  estimate  aircraft  system  test  and 
evaluation  costs  with  better  accuracy  and  in  greater  detail 
than  is  currently  feasible, 

NAVAIR  has  funded  this  project  to  collect  the  data  and 
develop  an  accurate  and  reliable  way  to  estimate  these  cost 
relationships.   The  key  problems  related  to  this  research 
will  involve  the  data  collection  from  the  defense 
contractors  and  the  standardization  of  their  data. 
Representatives  at  NAS  Patuxent  River,  Md.,  and  Edwards  Air 
Force  Base  Flight  Test  Center  have  already  stated  that  they 
will  assist  in  the  collection  of  data  at  their 
installations . 


C.   AREA  OF  RESEARCH 

Estimation  of  cost  drivers  in  Aircraft  System  Test  and 
Evaluation  (ASTE);   conceptual  modeling  and  some  empirical 
evidence.  (Air  Force,  Navy,  and  civilian  contractors — 
subject  to  data  availability) 


D.   RESEARCH  QUESTIONS 

1,  What  are  the  cost  drivers  in  ASTE? 

2.  How  do  the  characteristics  and  complexity  of  the 
aircraft  relate  to  the  average  engineering  hour? 
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3.  What  statistical  and  econometric  models  can  be 
appropriately  applied  to  ASTE?   Is  multiple  regression 
appropriate  to  this  problem? 

4.  Are  the  military  work-breakdown  structures  pertinent  to 
ASTE? 

5.  How  can  a  computer  system  (hardware  and  software)  be 
designed  and  implemented  to  conduct  testing  of  the 
models? 

6.  How  have  data  been  defined  and  collected  in  different 
test  sites  (military  and  civilian)?   Can  they  be 
filtered,  and  standardized?   What  extrapolation  method 
should  be  used  for  missing  data? 

7.  How  do  these  cost  drivers  affect  the  average 
engineering  hour  per  hour  of  flight? 

8.  Is  there  any  correlation  between  the  number  of 
instrumentation  channels  and  the  number  of  engineering 
hours? 

9.  Is  there  a  linear  cost  relationship  between  different 
types  of  aircraft  with  respect  to  their  complexity  and 
characteristics? 

10.  How  sensible  and  stable  are  the  elasticities  of  the 
cost  drivers? 

11.  What  are  the  implications  of  the  findings? 

12.  Where  is  it  most  cost-effective  to  conduct  future 
military  ASTE — at  the  contractors  facilities  or  at  the 
established  flight  test  centers?   Is  the  answer  the 
same  for  both  flight  test  centers? 

13.  Do  the  findings  have  an  impact  on  the  service's 
operational  test  and  evaluation? 

14.  How  does  inflation  affect  the  predictive  power  of  the 
findings? 


E.   SCOPE  OF  THE  RESEARCH 

Develop  essentially  parametric  cost  estimating  relationships 
for  selected  system  test  and  evaluation  elements  and  each 
appropriate  functional  cost  category  by: 

1.   Defining  a  sample  of  pertinent  aircraft  problems. 
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2.  Formulating  a  work  break-down  structure/functional  cost 
elements  matrix. 

3.  Acquiring  historical  cost  data, 

4.  Organizing  the  data  for  analysis. 

5.  Employing  statistical  methods  to  develop  cost 
estimating  relationships. 

6.  Documenting  data,  sources,  rationale,  and  methodology. 

F.   METHODOLOGY 

This  research  seeks  to  apply  microeconomic  theory  and  cost 
analysis  in  the  area  of  Aircraft  Systems  Test  and  Evaluation 
(ASTE).   Multiple  linear  and  logarithmic  regression  are 
considered  as  analytical  tools.   This  research  also  applies 
the  concepts  defined  in  system  analysis  and  design  to 
implement  a  decision  support  software  for  ASTE. 

Interviews  with  defense  contractors,  Patuxent  River  Naval 
Test  Flight  Center,  Edwards  Air  Force  Base  Flight  Test 
Center,  and  on-site  data  collection  are  envisaged  for  the 
first  part  of  the  research.   Utilization  of  statistical 
software  packages  to  identify  elasticities  of  cost  drivers 
will  constitute  the  second  part  of  the  research. 


I.   BENEFITS  OF  STUDY 

There  is  an  increasing  requirement  from  high  levels  that  the 
DOD ' s  aircraft  cost  estimators/analysts  provide  explicit 
estimates  for  the  sub-elements  of  aircraft  systems  test  and 
evaluation  efforts.   This  research  project  will  attempt  to 
develop  cost  estimating  relationships  for  test  and 
evaluation  elements  used  for  aircraft  systems.  The  results 
of  this  project  will  benefit  DOD,  Patuxent  River  Naval  Test 
Flight  Center,  Edwards  Air  Force  Base  Flight  Test  Center, 
and  civilian  contractors. 
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APPENDIX  B:  NAVAIR  PROJECT  POINTS  OF  CONTACT 


Air  Force  Flight  Test  Center 

Mrs.  Carol  Gutherie  805-227-3710 

Code  AC 

Edwards  Air  Force  Base,  Ca.   93523 

Boeing  Aerospace  Company 

Mr.  Henry  F.  Schwartz  206-655-3244 

M/S  1758 

P.O.  Box  3707 

Seattle,  Wa.   98124 

Douglas  Aircraft  Company 

Mr.  Len  Chartier  213-593-0920 

Cl-243  18A-40 

3855  Lakewood  Blvd. 

Long  Beach,  Ca.   90846 

Douglas  Aircraft  Company 

Mr.  Loren  Frye  213-593-4745 

Cl-240  41-80 

3855  Lakewood  Blvd. 

Long  Beach,  Ca.   90846 

Fairchild  Republic  Company 

Mr.  Stan  Granowetter  516-531-3251 

Conklin  Street 

Farmingdale,  N.Y.   11735 

Fairchild  Republic  Company 

Mr.  Robert  Hoffman  516-531-2752 

Building  55 

Conklin  Street 

Farmingdale,  N.Y.   11735 

General  Dynamics  Corporation 

Mr.  John  Denheyer  817-777-1581 

M/Z  1638 

P.O.  Box  748 

Fort  Worth,  Tx.   76101 

Grumman  Aerospace  Corporation 

Mr.  Jim  McDonagh  516-575-5158 

MS  C20-GHQ 

Bethpage,  N.Y.  11714 
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Lockheed-Georgia  Company 

Mr.  Bob  Farmer  404-424-2741 

Dept.  81-11,  Zone  23  ■ 

86  South  Cobb  Dr. 

Marietta,  Ga .   30063 

Lockheed-California  Company 

Mr.  Phil  Finkle  818-847-3343 

Dept.  81-16,  B/66,  P/A-1 

P.O.  Box  551 

Burbank,  Ca .   91520 

LTV  Aerospace  and  Defense 

Aero  Products  Division 

Mr.  Clint  Miller  214-266-8755 

MS  220-70 

P.O.  Box  225907 

Dallas,  Tx.   75265 

McDonnell  Aircraft  Company 

Mr.  Raymond  Yarck  314-234-7100 

Dept.  015 

P.O.  Box  516 

St.  Louis,  Mo.   63166 

Naval  Air  Systems  Command 

Mr.  Bob  Patterson  202-692-3935 

AIR-5243 

Jefferson  Plaza  2 

Washington  D.C.   20361 

Naval  Air  Test  Center 

Karen  Tyson  301-863-3691 

Business  Resource  Department 

Antisubmarine  Aircraft  Test  Directorate 

Patuxent  River,  Md.   20670-5304 

Rockwell  International  Corporation 

Mr.  Gayle  Turner  614-239-2796 

Bldg.  3 

4300  E.  Fifth  Ave. 

Columbus,  Oh.   43216 

Rockwell  International  Corporation 

Mr.  George  Bostater  213-414-4819 

Dept  196,  011-ZR09 

P.O.  Box  90098 

Los  Angeles,  Ca.   90009 
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APPENDIX  3:  CONTRACTOR  FORM  LETTER  #2 


December  5,  1986 

Navair  Research  Team 
Code  54Bd 

Naval  Postgraduate  School 
Monterey,  Ca.  93943 

We  would  like  to  thank  you  again  for  the  opportunity  we  had 
to  meet  with  you  and  visit  your  facilities  to  discuss  our 
research  project  last  month.   Your  inputs  were  extremely  valuable 

in  giving  us  an  overview  of  's  methods  used  for  the  cost 

estimation  of  aircraft  systems  and  your  test  and  evaluation 
cprograms.   At  that  time,  we  promised  to  clarify  our  data 
collection  requirements  following  a  meeting  with  our  Naval  Air 
Systems  Command  (NAVAIR)  project  sponsor  on  20-21  November.   The 
information  you  provided  was  discussed  and  helped  focus  the  scope 
of  our  project  and  the  specific  areas  of  data  required  for  our 
research. 

The  scope  of  this  project  will  initially  include  only  the 
following  areas  of  aircraft  system  test  and  evaluation: 

Contractor  Flight  Test 
Wind  Tunnel  and  Article  Test 
Static  Article  and  Test 
Fatigue  Article  and  Test. 

In  our  initial  letter,  we  provided  a  matrix  of  Work  Breakdown 
Structure  (WES)  sub-elements  in  these  areas  and  a  listing  of  the 
following  direct  labor  hour/dollar  cost  categories  pertaining  to 
these  sub-elements: 

Engineering  Hours 
Manufacturing  Hours 
Tooling  Hours 
Quality  Control  Hours 
Logistic/Product  Support  Hours 
Material  Dollars 
Total  Dollars. 

The  basic  System  Test  and  Evaluation  data  required  for  our 

project  are  direct  labor  hours  and  dollar  values  for  the  

aircraft  system(s)  your  facility  produced,  broken  down  into  the 
sub-element  matrix,  and  also  listed  in  a  time-phased  fashion. 

In  order  to  normalize  this  information  between  contractors, 
it  is  important  that  we  know  your  specific  definition  of  the  WBS 
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sub-elements  used  by  your  company  for  direct  labor  hour  tracking. 
For  the  direct  labor  hours,  we  request  that  you  provide  a  brief 
description  of  how  the  engineering,  manufacturing,  tooling, 
quality  control  and  support  hour  categories  were  derived  if  your 
records  differ  from  the  Contractor  Cost  Data  Reporting  methods. 

For  the  aircraft  flight  test  portion  of  our  research,  we 
request  the  number  of  aircraft  used  in  your  flight  test  program 
with  associated  flight  hours. 

We  request  the  data  for  direct  labor  hour/dollar  categories 
and  the  flight  test  hours  be  expressed  in  a  time-phased  fashion. 
If  this  data  is  not  available  from  your  records  time-phased, 
please  estimate  the  start  and  completion  dates  and  indicate  the 
peak  activity  levels  within  the  test  period. 

A  cost  driver  indicated  for  the  flight  test  program  was  the 
total  number  of  separate  tests  to  be  conducted  during  the  flight 
phase.   If  you  have  information  available  to  indicate  the  total 
number  of  separate  tests  and  a  brief  description  of  these  test 
categories,  it  will  help  in  the  quality  of  our  final  product. 

We  feel  that  this  research  can  be  valuable  to  our  military 

project  sponsor  and  also  useful  to  as  a  cost 

estimating  tool.   The  accuracy  of  our  final  product  will  depend 
primarily  on  the  data  we  receive  from  your  company  and  our 
ability  to  correlate  it  with  the  data  from  the  other  contractors 
we  have  visited. 

We  realize  the  constraints  of  your  busy  schedule,  but  would 
appreciate  to  be  able  to  receive  as  much  of  this  data  as  possible 
by  January  10,  1987  to  be  used  in  a  meeting  which  will  be  held 
with  the  NAVAIR  project  sponsor  January  15,  16  and  17, 

We  again  express  our  thanks  for  the  assistance  that  you  have 
provided  and  your  continued  interest  in  our  program.   We  will 
call  you  next  week  and  attempt  to  answer  questions  that  you  may 
have  concerning  this  data  format  or  limitations  that  may  be 
peculiar  to  a  specific  aircraft  system. 

Sincerely, 
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The  following  are  to  be  broken  down  by: 

1.  Engineering  hours 

2.  Manufacturing  hours 

3.  Tooling  hours 

4.  QC/ILS/Product  support 

5.  Material  $ 

6.  Total  $ 

CONTRACTOR  FLIGHT  TEST 
INSTRUMENTATION 
FLIGHT  AND  DEMONSTRATION 
FUNCTIONAL  GROUND  CHECKOUT 
FLIGHT  TEST  AND  DEMONSTRATION 
FLIGHT  TEST  SUPPORT 

FLIGHT  TEST  SPARES  &  REPAIR  PARTS 

SUPPORT  EQUIPMENT 

CONTRACTOR  TECHNICAL/MAINTENANCE  SERVICES 

WIND  TUNNEL  ARTICLE  &  TEST 
WIND  TUNNEL  ARTICLE 
WIND  TUNNEL  TEST 

STATIC  ARTICLE  &  TEST 
STATIC  ARTICLE 
STATIC  TEST 

FATIGUE  ARTICLE  &  TEST 
FATIGUE  ARTICLE 
FATIGUE  TEST 


Other  breakdowns : 

1.  #  of  flight  test  a/c  and  first  flight  of  each. 

2.  Flight  hours  (  time  phased) 

3.  Wind  tunnel  occupancy  hours  (  time  phased) 

4.  Type  of  wind  tunnel  test 

5.  #  of  instrumentation  channels  for  instrument  flight  test 
A/C. 
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